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ABSTRACT 


The volume of data available to military decision makers is vast. Leaders need tools to 
sort, analyze, and present information in an effective manner. Software complexity is 
also increasing, with user interfaces becoming more intricate and interactive. The Data 
Visualization Tool (DaViTo) is an effort by TRAC Monterey to produce a tool for use by 
personnel with little statistical background to process and display this data. To meet the 
program goals and make analytical capabilities more widely available, the user interface 
and data representation techniques need refinement. This usability test is a task-oriented 
study using eye-tracking, data representation techniques, and surveys to generate 
recommendations for software improvement. Twenty-four subjects participated in three 
sessions using DaViTo over a three-week period. The first two sessions consisted of 
training followed by basic reinforcement tasks, evaluation of graphical methods, and a 
brief survey. The final session was a task-oriented session followed by graphical 
representations evaluation and an extensive survey. Results from the three sessions were 
analyzed and 37 recommendations generated for the improvement of DaViTo. 
Improving software latency, providing more graphing options and tools, and inclusion of 
an effective training product are examples of important recommendations that would 
greatly improve usability. 
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I. 


INTRODUCTION 


A. PURPOSE 

In the early years of computers and the software to use and operate them, most 
applications were developed by highly trained and specialized personnel. Software was 
designed for a user with commensurate specialized skills in the computing arena. Jobs 
that required the use of computers also required specific training or education in order to 
use them. As computing power began to increase and the use of computers spread across 
professions and into homes, there were more users with little or no education or training 
in the field. Software development evolved with this, merging toward a central theme of 
usability for the end user. This enabled computers and software to traverse into all fields 
of work for a full range of users. Usability has many definitions but in software 
engineering it is “the quality of a system with respect to ease of learning, ease of use, and 
user satisfaction” (Rosson & Carroll, 2002). These three usability perspectives are 
directly reflected in a summary statement of the goals TRAC Monterey stated when 
developing the Data Visualization Tool (DaViTo). Their intention was to develop an 
open source tool that would be easy to use for someone unfamiliar with statistical 
methods to produce useful visualizations and analysis of spatial-temporal data, and be 
able to present that output in a way that could be understood by ground commanders 
(Evangelista, Henry, Buttrey, & Whitaker, 2012). The overall purpose of this thesis is to 
determine how DaViTo in its current state needs to be improved to meet the usability 
goals it was originally developed to fulfill. 

B. OBJECTIVES 

The goal is to provide concrete and substantive recommendations to improve the 
usability and workflow of the DaViTo software. Research for this thesis began with two 
main objectives, both with subsets of secondary objectives. 

The primary objective consists of evaluating the usability and efficiency of the 
DaViTo to determine if the workflow to complete a task matches how a user with 
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minimal training thinks the workflow should be carried out. The following subset of 
secondary objectives was used to complete the first primary objective. 

• Identify speed and accuracy to complete tasks by investigating mouse 
clicking trends, task completion times, and number of correct answers. 

• Identify misunderstandings of the software interface such as button 
symbology, names, mouse-overs, or locations. 

• Utilize eye-tracking characteristics such as gaze patterns and dwell times 
to identify software inefficiencies. 

The second objective is to identify areas where efficiency and usability can be 
improved, and translate these into actionable recommendations. Again, a subset of 
secondary objectives was used to complete the primary objective. 

• Pinpoint ways to improve the software’s visual palette in a manner that 
will increase information transfer to users. 

• Identify button improvements such as location and identification that will 
increase ease of use. 

• Identify ways to optimize input device use to increase efficiency. 

The final objective was to determine if there were deficiencies in the use of data 
representation techniques within DaViTo, and how to correct them. The secondary 
objectives to complete this were as follows. 

• Identify graph types that best represent the statistical analysis of typical 
data sets used in DaViTo. 

• Determine additional graphing tools needed to improve output statistical 
analysis by users with little or no statistical training. 

C. SCOPE AND LIMITATIONS 

This thesis is limited to the usability of the DaViTo user interface itself. There is 
no intention to investigate the software behind DaViTo despite one of the software goals 
being to allow user modification of the code for customization to individual needs. 
DaViTo is a broad program that draws from three different open source software 
packages, R, JFreeChart, and OpenMap. Additionally, the goal was to test the usability 
of new users rather than analysts, so advanced functionality of the software such as 
outputting data to R for advanced statistical analysis is not evaluated. Due to time 
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constraints study participation is limited to 24 subjects at approximately three hours each. 
This sample size is large enough to thoroughly evaluate resultant data and sufficient for 
potential loss or corruption of individual data sets, yet small enough to be tested within 
research lab and personnel time restrictions. 

Eye-tracking is a key component of the usability testing of this thesis, and is not 
known to have been frequently used previously in this area of research. While the eye¬ 
tracking itself and the statistics associated with it are of interest, the scope of this thesis 
limits the use of eye-tracking results. They were used solely for developing 
recommendations for software improvement in the areas of workflow analysis and 
usability. There is a large amount of data available from the EyeWorks software such as 
blink data, saccades, and cognitive load based on pupil contraction. Eye-tracking was 
only a component of the data evaluated and as such only scan patterns and fixations were 
used in the end. The outcome of the eye-tracking performance is not evaluated outside of 
these areas. 

D. THESIS ORGANIZATION 

Chapter I - Introduction. This chapter describes the purpose and overall 
objectives of the thesis, as well as identifies the research scope and limitations. 

Chapter II - Background. This chapter introduces and explains the DaViTo 
software and usability. Additionally, the purpose of usability testing generally and with 
respect to DaViTo is broken down. 

Chapter III - Methodology. The experimental design is discussed in depth, to 
include training and tasks, subject participation, and the actual experimental procedure. 

Chapter IV - Results and Data Analysis. This chapter presents results from the 
collected data and explains how the various sources are analyzed. The findings of the 
data analysis are also discussed. 

Chapter V - Recommendations. This is the consummation of the research into 
recommendations for change and improvement of the software to improve its usability. 
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This is input for TRAC Monterey as the software continues to be improved, and is broken 
down into four categories: recommendations based on task results, graph results, survey 
results, and author recommendations. 

Chapter VI - Conclusion and Future Work. The final chapter will summarize the 
results and recommendations, and detail future recommended work. 

Appendix A - Consolidated Recommendations. This appendix contains all 
37 recommendations to TRAC Monterey in one table. 

Appendix B - Additional Task Results. This appendix contains additional results 
from the task analysis portion. 

Appendix C - Additional Graph Results. This appendix contains additional 
results from the graph analysis portion. 

Appendix D - Additional Survey Data. This appendix contains additional results 
from the survey data. It is divided into the three common questions for all three sessions’ 
data and the final survey, and contains all subject free-form input. 

Appendix E - Task Worksheet. This appendix contains the task list worksheet 
used by the experiment proctor for the final session, which lists all 11 tasks. 

Appendix F - Approved Institutional Review Board (IRB) Documents, Including 
the Call for Participants, Consent Form, Demographic Survey, and Final Survey 
questions. This appendix contains the approved Institutional Review Board Protocol 
cover sheet and author-generated documents for the usability test. 
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II. BACKGROUND 


A. WHAT IS DA VITO 

The Data Visualization Tool (DaViTo) was developed by TRAC Monterey to 
give U.S. Army personnel a data exploration tool that could be used to discover patterns 
and conduct analysis of the huge amounts of data that are generated in today’s 
sophisticated military operational environments. From the DaViTo project report: 

DaViTo specializes in the exploration of point process data that contains 
both a spatial and temporal component. Pre-loaded with simple yet novel 
data visualization techniques, DaViTo can enhance the analyst’s ability to 
understand point processes across time and space. DaViTo provides an 
interface that allows users to tessellate geographic areas, define unique 
geographical areas, or load predefined geographic polygons. Once 
geographic areas of interest have been defined, DaViTo provides a suite of 
data exploration methods that can be applied to the data that resides within 
these areas. Since DaViTo is government owned software built solely 
with open source software, the potential of this software is limited only by 
the skills and expertise of the user and developer community. 
(Evangelista, Henry, Buttrey, & Whitaker, 2012) 

DaViTo was developed with four goals in mind for the software. One was that it 
be capable of producing useful visualizations and analysis of data with a spatial-temporal 
component. This is an essential part of the program, as it was desired to convert the vast 
amounts of data into something that can be interpreted. This leads into a second goal, 
representing the data with statistical and graphical output that can be grasped and 
understood by ground commanders. The importance of this was to allow them to take the 
data input to the tool, transform that data into a visual representation that shows the 
points of significance, and present it to decision-makers in a logical and comprehensible 
manner. A third goal seen as very important to TRAC Monterey was to distribute the 
software as open source, created from open source programs. The point in this goal was 
to give users the power to modify the tool to meet their specific needs, providing an 
instrument that can be continually adapted by the development community. The final 
goal was that the software would be “easy to use for someone unfamiliar with statistical 
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methods and Geographic Information Systems (GIS)” (Evangelista, Henry, Buttrey, & 
Whitaker, 2012). This goal was the driving force for the conduct of this thesis. TRAC 
Monterey’s approach was to conduct a usability test of the software package, after its 
initial release, to give recommendations for improvement that could be applied to a future 
revision of DaViTo that would better meet this goal. It was identified that by making the 
tool more usable it could be utilized further, beyond the use of an Army analyst, such as 
in deployed environments, and potentially by other branches of the armed forces. 

DaViTo is an open source program that is actually a compilation of three open 
source software packages. The open source aspect was a goal not only to allow for 
customization, but also for fiscal reasons. Similar commercial systems such as ArcGIS 
are very powerful, but widespread use is not possible due to the price. Additionally, 
commercially available products have the ability to perform some of the tasks that 
DaViTo does; however, none provide the merger of functionality that DaViTo possesses. 
As such, DaViTo was developed using three open source software tools. 

The first, OpenMap, developed by Raytheon subsidiary, BBN (Bolt, Beranek, and 
Newman) Systems, is “a Java Beans based toolkit for building applications and applets 
needing geographic information” (Raytheon BBN Technologies, 2005). OpenMap is the 
GIS portion of DaViTo that provides the mapping capability. It is the root of the 
interface for users, allowing interaction with the map and display of data on it. Using 
OpenMap, users have the ability to load and manipulate various Shapefiles, datasets, and 
maps in multiple formats. 

JFreeChart was the second open sources software tool implemented in DaViTo. 
JFreeChart was developed by David Gilbert of Object Refinery Fimited, and is a widely 
used Java chart library. It allowed the developers of DaViTo to incorporate professional 
quality data representations into the project. The ability to produce high quality visual 
output was necessary for both display when using the software and for import to planning 
briefs and training. 

Statistical analysis was incorporated into DaViTo with the use of the R Project for 
Statistical Computing. This open source software, known as R, “is a language and 
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environment for statistical computing and graphics. R provides a wide variety of 
statistical and graphical techniques, and is highly extensible” (The R Foundation for 
Statistical Computing). By integrating R into DaViTo, the ability to conduct complex 
statistical analysis is given to the user, and is only limited by their skill level. 

B. WHAT IS USABILITY 

When the term usability is brought up, its actual meaning depends upon the 
context within which it is used. Common definitions from the Merriam-Webster 
Dictionary are “capable of being used” or “convenient and practicable for use” (Merriam- 
Webster, Incorporated, 2012). In the context of computer software development and 
testing it has a much more specific definition. In Rosson and Carroll’s Usability 
Engineering, usability is defined as “the quality of a system with respect to ease of 
learning, ease of use, and user satisfaction” (Rosson & Carroll, 2002). It is even more 
generally described in The Handbook of Usability Testing as “what makes something 
usable is the absence of frustration in using it” and further defined as “when a product is 
truly usable, the user can do what he or she wants to do the way he or she expects to be 
able to do it, without hindrance, hesitation, or questions” (Rubin & Chisnell, 2008). Both 
of these definitions apply to the purpose of this thesis research. With the overall goal of 
developing recommendations for improvements to the usability of the DaViTo software, 
clearly improving the ease of learning, ease of use, and user satisfaction, while 
minimizing user frustration, would be drivers for these recommendations. Throughout 
the development of the experiment and subject trials frustration was frequently seen with 
the software, pointing to areas where improving the ease of use would improve 
satisfaction. 

C. PURPOSE OF USABILITY TESTING 

The purpose of usability testing in the context of this thesis is to test the usability 
performance and workflow functionality of the DaViTo software. The development of 
this experiment was driven in large part by definitions, designs, and procedures outlined 
in two references. Rosson and Carroll cite usability professionals from Digital 
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Equipment Corporation describing usability engineering as “using the term to refer to 
concepts and techniques for planning, achieving, and verifying objectives for system 
usability. The key idea is that measurable usability goals must be defined early in 
software development, and then assessed repeatedly during development to ensure that 
they are achieved” (Rosson & Carroll, 2002). While TRAC Monterey developed goals 
for the development of the DaViTo software, it did not follow this definition with respect 
to end user usability. The Handbook for Usability Testing refers to usability testing in 
the same manner in which it was desired to test DaViTo in this post-development 
method. The authors deem it “a process that employs people as testing participants who 
are representative of the target audience to evaluate the degree to which a product meets 
specific usability criteria” (Rubin & Chisnell, 2008). Again, throughout the development 
of DaViTo, no subject testing was conducted to refine the software and ensure the stated 
goal of “easy to use for someone unfamiliar with statistical methods and Geographic 
Information Systems” (Evangelista, Henry, Buttrey, & Whitaker, 2012). Providing 
recommendations that would aid TRAC Monterey in meeting this goal in future releases 
of the DaViTo product was the purpose of this usability test. 

As the thesis was conceived and the experiment methodology began to be 
formulated, it was necessary to break usability into several attributes, most of which were 
derived from The Handbook of Usability Testing (Rubin & Chisnell, 2008). The 
attributes are usefulness, efficiency, effectiveness, leamability, and satisfaction, which 
are described in the matrix in Table 1, and provided the basis for the different sections of 
the surveys developed for the experiment, contained in Appendix E. In addition to the 
five attributes identified by Rubin and Chisnell, categories for improvements and 
visualizations were used as well. 
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Attribute 

Description 

Usefulness 

The degree to which a product enables a user to achieve his or her 
goals, and is an assessment of the user’s willingness to use the 
product at all. 

Efficiency 

The quickness with which the user’s goal can be accomplished 
accurately and completely and is usually a measure of time. 

Effectiveness 

The extent to which the product behaves in the way that users expect 
it to and the ease with which users can use it to do what they intend. 

Learnability 

A part of effectiveness and has to do with the user’s ability to 
operate the system to some defined level of competence after some 
predetermined amount and period of training. 

Satisfaction 

Refers to the user’s perceptions, feelings, and opinions of the 
product. 

Improvements 

Raw recommendations from the subjects with regard to ways to 
improve the software. 

Visualizations 

Refers to the user’s interpretation of the user interface and used to 
determine if it is inherently usable in its current state. 


Table 1. Matrix of usability attributes and their descriptions (Rubin & Chisnell, 2008). 
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III. METHODOLOGY 


A. OVERVIEW AND OBJECTIVES 

The overall objective of the experiment was to collect objective and subjective 
data from participants using the DaViTo software that could then be transformed into 
actionable recommendations for improvements to the DaViTo software. Data collection 
was conducted through a number of different methods. The first was through a 
demographic survey contained in Appendix F. Another method was a set of 
questionnaires at the conclusion of each session, with the final questionnaire being very 
extensive, also contained in Appendix F. The questionnaires contained questions with 
selectable multiple choice responses as well as fill in the blank where subjects could give 
free-form opinionated answers. Data was also collected by evaluating subject’s ability to 
evaluate graphical representations in an accurate and timely manner. The final method of 
data collection utilized automated eye-tracking to collect mouse clicks, reaction time, and 
gaze data such as fixations and scan patterns. 

Following the call for participants and subsequent scheduling, the experiment 
took place over three sessions. The first two sessions consisted of a brief training tutorial 
followed by some simple reinforcement tasks and questions. The objective of the 
tutorials was to present a method of software functionality to the participant on one 
screen, and for them to follow along on their own screen. The objectives of the final 
session were completion of more complex tasks, data representation questions, and an 
extensive questionnaire. 

B. EQUIPMENT 

I. Hardware 

The main hardware used for the experiment consisted of two laptops, one desktop, 
and a set of stereo cameras with an IR tracking pod. Additional hardware included 
multiple displays, a keyboard, and a mouse. For the final session the Gateway laptop was 
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removed because there was no video tutorial. The data was kept securely on the desktop 
computer then moved to a secure folder on the share drive for analysis. The desktop had 
the following specifications: 

• Operating System: Windows 7 Professional (64-bit) 

• Manufacturer: Hewlett-Packard (HP) 

• Model: HP 2800 Workstation 

• Processor: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz 

• RAM: 6.00GB 

• Screens: Two 24 Inch LCD Displays 

The laptop for tutorial playback had the following specifications: 

• Operating System: Microsoft Windows XP (32-bit) 

• Manufacturer: Gateway 

• Processor: Mobile AMD Sempron™ 3300-1- 1.99 GHz 

• RAM: 448 MB 

• DVD Player: Philips DVD-i-RW model SDVD8441 

• Screen: 16 in LCD display 

The laptop for faccLAB control had the following specifications: 

• System: Microsoft Windows XP (32-bit) 

• Manufacturer: Hewlett Packard 

• Processor: Intel(R) CoreTM2 CPU T7200 @ 2.00 GHz 

• RAM: .99 GB 

• Screen: 15 in LCD display 

The subjects utilized the following input devices: a Dell two-button mouse with a 
scrolling wheel in the center as well as a standard Dell QWERTY keyboard. 
Additionally, directly in front of the keyboard was a pair of stereo cameras and an IR pod 
that took input from the subject’s eyes. The IR pod and stereo camera set were 
manufactured by Seeing Machines; model number IRPD-PI-R2. 
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2 . 


Software 


Three software packages were used for the experiment. The program used to 
initiate and control subject eye-tracking was faceLAB, version 5.0.5. The experiment 
was designed and implemented using the EyeWorks software suite, version 3.7.3.145. 
Finally, the DaViTo software package was used by the participants and was the software 
of interest and evaluation 

Data was collected automatically via the eye-tracking software faceLAB and 
EyeWorks. FaceLAB was the software vehicle to track subject’s eyes and facial features. 
It was coupled with the EyeWorks software, which would receive the automated data 
from faceLAB, and record it for future analysis. The EyeWorks suite was the backbone 
of the experimental design, and is an all-inclusive research software package for users of 
eye-tracking. The software has three functional areas. Design, Record, and Analyze, all 
of which were used. The Design component was used to format the experiment from 
beginning to end for each session. All tasks, questions, and surveys were built into the 
Design shell. The Record component ran each session. The script was started at the 
beginning of each session, and within the Record window all tasks were prompted, 
answers collected, and surveys conducted. This division of the software package was 
also used to capture not only eye movement, but launch the experiment, partition portions 
of each session for ease of data analysis, and record mouse clicks. It was also useful for 
real time observation of the subjects view and where their gaze was directed during the 
experiment, as well as providing a video recording of every session. EyeWorks analyze 
was then used to provide heat maps and graphs of gaze data, display mouse clicks, and 
output survey answers to Excel. 

3. Physical Setup 

The arrangement of the physical hardware, shown in Figure 1, was identical for 
all three sessions. The exception was item 3 in Figure 1, the laptop used for the video 
tutorials in sessions one and two, which was removed prior to session three. The layout 
of all hardware components shown in Figure 1 are described in Table 2. 
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Controller 



Figure 1. 


Birds-eye-view diagram of the equipment configuration. Descriptions of 
each item are in Table 2. 


Number in 
Diagram 

Description 

1 

24 Inch LCD Screen - Displayed DaViTo software, tasks, and surveys 
that were created in Eye Works. 

2 

Secondary 24 Inch LCD Screen - Display for the experiment controller 
to monitor EyeWorks tracking data. 

3 

Gateway Laptop - Displayed the video tutorials. This was a standalone 
device and was removed prior to the final session. 

4 

Hewlett Packard Laptop - Controlled the faccLAB software. This was 
a standalone device facing the experiment controller. 

5 

Infrared and Stereo Camera Pod - Precision located to track the 
subject’s eyes for best results while not obscuring the subjects view. 

6 

Keyboard - Input device to respond to survey questions. 

7 

Mouse - Input device to allow the subject to interact with the DaViTo 
and EyeWorks software via the HP desktop. 


Table 2. Description of items in equipment configuration illustrated in Figure 1. 
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C. DESIGN OF THE EXPERIMENT 

1. Training Tutorial Development 

In order to standardize training for the DaViTo software and stay within time 
constraints, the training materials were designed as a recorded tutorial. Microsoft 
PowerPoint 2010, version 14.0.6123.5001, was used to generate slides showing the 
subjects what to do. The slides were then narrated and recorded using Camtasia Studio 
version 7. The testing script was developed using EyeWorks Design. Our Institutional 
Review Board (IRB) submitted time constraint was three one hour sessions. The goal 
was to keep both tutorial sessions less than 20 minutes each to allow ample time for task 
accomplishment, data representation questions, and questionnaire completion. They were 
also designed with ample breaks and pause points so subjects would not become lost in 
the tutorials. 

2. Questionnaire Development 

The final portion of each session was a series of questions drawn from the 
extensive list of questions submitted with the IRB package in Appendix F. The 
questionnaire portion of each session began with the same three questions. These 
questions asked the subject to rate the training for the first two sessions, how much more 
training they think is needed to use DaViTo in the field, and to list their general likes and 
dislikes. The final survey then had a much more extensive questionnaire consisting of 
the first three questions and then approximately 40 more questions derived from the 
usability factors matrix in Table 1. The questions were a mixture of multiple choice and 
free entry. These questions were designed to identify deficiencies with respect to the 
usability attributes, as well as elicit recommendations from the participant that would be 
directly targeted to each attribute. 
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3. 


Training Session One 


The first session was limited to software interface introduction. The video tutorial 
was exactly 18 minutes long. It began by introducing the thesis topic, the software 
designers, and the authors. Next, the purpose of DaViTo, its four goals, and six main 
functions were introduced to give the participants some background information 
pertaining to the software’s design functionality. Following the introduction, the user 
interface was introduced. The user interface was divided into five functional areas, as 
shown in Figure 2. Tutorial session one focused only on the Map Controls, Data Set, and 
Map and Data Display Area portions of the interface, with the Shapefiles and Display 
portions to be covered in session two. To begin, the Data Set section of the interface was 
introduced, and the participant was shown how to load various data sets into the software. 
They were then instructed on how to manipulate them, such as removing data sets or 
changing colors. This section was taught first to facilitate having a data set open in the 
software for the remainder of the tutorial. Once complete, instruction moved to the Map 
Controls portion of the DaViTo interface. Here, the selectable drop down menus and 
software control buttons were each introduced, continuing in the tutorial style with the 
participant following along on their own screen. To introduce the controls they were 
each discussed and then some were used to manipulate the Map and Data Display Area. 
Pause points were also inserted, and the participant was informed that they could pause 
the tutorial and take additional time to explore the functionalities just discussed if they 
desired. 
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Figure 2. Interface introduction slide from Tutorial Session One. This slide shows the 
breakdown of the DaViTo interface into five functional areas. 


4. Training Session Two 

The second session video tutorial began with a brief review of the initial session, 
followed by more advanced instruction. This second tutorial was slightly shorter at only 
15 minutes and 45 seconds. Following the introduction, a brief review of the Map 
Controls, Data Set, and Map and Data Display Area portions of the DaViTo interface was 
conducted, with an opportunity for the user to pause the tutorial to interact and refresh 
themselves with any portions of the interface that they felt necessary. Once complete, the 
focus shifted to employing Shapefiles, manipulating statistical output, and conducting 
statistical analysis, all within the Map and Data Display Area. While this required use of 
the previously covered Map Controls and Data Set areas of the interface, it made frequent 
use of the Shapefiles and Display regions. The participants then loaded a similar dataset 
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to the previous session to work with through the first several minutes of the tutorial. The 
first new technique taught was loading a Shapefile with predefined polygons, centering 
the interface on that area of the screen, and then viewing and renaming graphs populated 
in those polygons. The subjects were then instructed on methods for making user defined 
polygons for specific smaller regions of interest, and the accompanying statistical output. 
The next training item covered was the loading and use of map layers within the Map and 
Data Display Area, and another method for user defined polygons. Using the 
accompanying graphs, the remaining options for graph modification were covered. 

5. Session Three 

The third and final session included no training; rather it was focused on 
conducting a sequence of tasks. There were several purposes for the last session. The 
first was to observe the workflow of the participants as they conducted tasks. This was 
done primarily via eye-tracking with the EyeWorks software, and by the administrators 
of the experiment recording subject’s answers to the final two tasks. The complete list of 
tasks for session three is contained in Appendix E. It began with tasks one through eight 
as a series of step-by-step simple tasks such as navigating to Africom and Iraq, loading 
data and Shapefiles, and creating user-defined polygons. The intent was to refresh the 
software and functionality of the interface. The ninth task removed all current data sets 
and Shapefiles, and then loaded the sets for the final two tasks. Task ten was a more 
complex and comprehensive task that required the participant to work through several 
steps without having the step-by-step questioning entered into the EyeWorks software to 
direct them through completion. The objective was for the subject to complete a task 
requiring a full series of actions from loading a data set to generating a graph that could 
be used to answer a theoretical commander’s question. The participant was directed to 
determine the total number of lED attacks for 2010 in the most northwestern provincial 
boundary of Afghanistan. Essentially, this task began with a blank slate and required 
loading a dataset and Shapefile, then navigating the Map and Data Display Area to 
Afghanistan by a means of their choice, and generating and opening a specific graph. 
Subjects were then required to answer the question about that graph. If a subject could 


18 



not complete the tasks, he or she was unable to answer the question. The final task was a 
trend analysis question that also required the completion of all of task number ten to 
answer. Answers to questions ten and eleven were recorded by the administrators of the 
experiment. Following the completion of task eleven, all work within DaViTo was 
complete. The participant then proceeded to the final series of graphing questions, with 
answers recorded by EyeWorks Record. The final portion of the session consisted of an 
extensive questionnaire of 50 questions composed from the seven usability factors. 

6. Experiment Scheduling Considerations 

The overall design of the experiment was for it to take place over a three week 
timespan for each participant, within a six week window of open scheduling available. 
This gave subjects the flexibility to choose a four week period in which to conduct their 
first session and still have two weeks following that for the remaining two sessions. 
Subjects were to schedule their sessions no less than six days and no more than eight days 
apart. The goal was seven days between, but plus or minus one day was allowed for 
flexibility. Finally, sessions were available throughout the Monday through Friday 
workweek, from 0700 to 1800 daily. 

D. SUBJECTS 

In accordance with the approved IRB, all subjects were taken from the pool of 
personnel of the Naval Postgraduate School (NPS). This included the student population, 
civilians, and foreign officers with only two conditions. The first condition was that a 
participant could not be colorblind due to the coloring of the charts used in DaViTo. The 
second required that the individual not be involved in the development of the DaViTo 
software. Participation in the study was completely voluntary and it was made clear 
during recruitment and throughout the experiment that the subjects were free to stop at 
any time, as well as the fact that there would be no compensation for participation. The 
demographic survey for the subject backgrounds is contained in Appendix F. 

The experiment sessions were scheduled to be conducted over a six week period, 
on weekdays between 0700 and 1800. When scheduling their three sessions, subjects 
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were directed to sign up for one hour blocks in three consecutive weeks, preferably seven 
days apart, but within a window of six to eight days between sessions. Almost all 
subjects signed up for sessions with seven days of separation between them, and all 
sessions were scheduled with the six to eight day gap between sessions. Every subject’s 
sessions were completed in five weeks and the last week allotted for experiments did not 
need to be utilized. 

The experiment was limited to 24 subjects in order to have an adequate amount of 
data for statistical analysis, but to also have enough time to conduct 72 hours of sessions. 
Recruitment was conducted through mass e-mail, word of mouth, and via classroom 
briefs. The IRB approved recruitment document is contained in Appendix F. The goal of 
24 subjects was met through the recruitment efforts, with a fairly diverse pool. The 
average participant age, as shown in Table 3, was 34.92, consisted almost entirely of O- 
3s and 0-4s, presented in Table 4, and the dominant service, seen in Table 5, was the 
U.S. Navy. The demographic survey responses also permitted the subjects to assess their 
computer skill level and average usage. Nearly one-third of subjects rated their skill level 
at intermediate, as seen in Table 6, while Table 7 shows that half the subjects indicated 
usage in excess of 21 hours per week. Other items of note from the demographic survey 
include a majority of subjects having a preference for the Windows operating system, 
shown in Table 8, seven subjects had some unspecified amount of previous experience 
with the mapping and spatial analysis tool ArcGIS, and no subjects had any knowledge of 
the DaViTo software. 


Subjects Age (years) 

Min 

28 

Max 

42 

Average 

34.92 

Mode 

40 

Median 

35 

Standard Deviation 

4.57 


Table 3. Breakdown of subject population by age. 
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Number of Subjects 
By Rank 

02 

0 

03 

11 

04 

12 

05 

1 

06 

0 


Table 4. Breakdown of subject population by rank. 


Number of Subjects By 
Service 

Army 

7 

Navy 

12 

Marine Corps 

4 

Air Force 

0 

Retired 

0 

International 

1 


Table 5. Breakdown of subject population by rank. 


Computer Skill Level 

Beginner 

1 

Novice 

6 

Intermediate 

15 

Expert 

2 


Table 6. Subjects’ assessment of their own computer skill level. 
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Subjects Hours Per Week 

Using a Computer 

(1) Hours 0-5 

1 

(2) Hours 6-10 

0 

(3) Hours 11-15 

5 

(4) Hours 16-20 

6 

(5) Hours 21+ 

12 

Average Response 

4.17 

Mode 

5 

Median 

4.5 

Standard Deviation 

1.03 


Table 7. Subjects reported weekly computer usage. 


Preferred Operating Sysl 

tern 

Mac 

4 

Windows 

19 

Other 

1 


Table 8. Summation of subject populations preferred operating system. 


E. EXPERIMENT PROCEDURE 

The experiment consisted of three sessions, and the procedure was common for 
all three sessions. The difference between sessions was content as covered previously. 
For the subjects, there was no difference between experimental setup and procedure 
except for in sessions one and two they controlled the playback of the tutorial, whereas in 
session three there was no tutorial. 

Prior to use each day it was necessary to check the calibration of the eye-tracking 
cameras and software. If required, a full system calibration was conducted. 
Additionally, stereo and IR camera positions were verified to be correct. DaViTo was 
then started and checked for proper function in case of any system freezes. 

Upon arrival for session one, subjects were directed to read the Consent to 
Participate in Research form, contained in Appendix F, and to sign and date it if in 
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agreement. The form was then signed by the researcher as a witness. Following that, 
they were directed to complete a demographic questionnaire, contained in Appendix F. 
The researcher instructed the subject to sit in front of the cameras and screens in a 
comfortable position. They were then linked up with faceLAB and eye-tracking 
commenced, stepping through the steps of the software to properly track facial 
components such as the corners of the eyes and mouth in order to optimize the tracking 
performance. FaceLAB tracking was then checked by following a series of dots on the 
screen. This data was saved by taking a screenshot and recording it on the experiment 
checklist. Once complete, control was then shared with EyeWorks and an additional dot 
check was done by having the participant follow dots on the screen. This completed 
setup of the participant with the eye-tracking software. Before beginning the experiment, 
the subject was given final instruction on how to use the left hand mouse for controlling 
the tutorial on the left screen, and how to use the right hand mouse for DaViTo and the 
EyeWorks software that controlled the experiment when moving from DaViTo to graph 
and survey questions. 

The actual experiment was now ready to begin. The experiment proctor would 
push record on the EyeWorks interface and instruct the participant to begin. Eor sessions 
one and two, the participants were directed within the EyeWorks window to follow along 
with the tutorial, and it would then exit out to the DaViTo interface. Eor these sessions, 
upon completion of the tutorial they would go back to the EyeWorks window. The 
subject would then proceed through the graphing and survey questions until complete. At 
this time recording in EyeWorks would be stopped, the data saved, and the subject 
reminded of their future session times. Similarly, for the final session the experiment 
would begin in the EyeWorks window with the initial task. The participant would 
proceed through all of the tasks, transition to graphing questions, and conclude with a 
longer series of survey questions. The subject was then thanked for their participation 
and dismissed. The proctor saved the information under the participant’s number and 
proceeded to setup the system for the next session. 
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F. DATA COLLECTION AND ANALYSIS STRATEGY 

1. Eye-Tracking Data 

This data collection was method was used during all three sessions. The eye¬ 
tracking data that was of the highest importance was that of session three, particularly the 
conduct of completing the tasks. Task completion for the first two sessions, as well as 
graph evaluation by the subjects was also of importance. 

While there are a large number of analysis techniques available through 
EyeWorks Analyze, only a few were used with the goal of delivering recommendations 
to TRAC Monterey for improvements to DaViTo. Raw mouse click data collected for 
each subject and session was used extensively, particularly in the final session for 
comparison to an efficiency baseline. Subject time to complete tasks was used for 
comparison to a time baseline. Additionally fixation data was utilized in the form of heat 
maps to determine visual concentration areas of the DaViTo interface. 

2. Graphs 

Data representation research was important to see if the graphical methods 
employed by default in DaViTo are the appropriate way to display data, as well as 
evaluate some similar methods of data representation. The current default is the line 
chart format, with the stacked bar format also available, but there are a myriad of 
possibilities with the inclusion of JFreeChart and the R statistical package. These include 
but are not limited to the stacked line with markers format, the stacked area format, and 
the clustered columns format, as well as many others. To evaluate this, at the conclusion 
of each series of tasks the participant was then directed within the EyeWorks Record 
session to answer similar questions about a number of graphs. Their answers were saved 
by EyeWorks Record. 

Participant answers were later imported to Excel and used in multiple analyses 
with Excel and IMP. The strategy was to evaluate the resultant data for efficiency, with 
time to complete and correctness the parameters of interest. 
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3. 


Questionnaires 


The participant answers to the surveys were recorded by EyeWorks Record, and 
then imported to Excel for analysis. The multiple choice questions were evaluated on the 
basis of average, mode, median, and frequency. The free entry questions were evaluated 
for similar responses and used to formulate participant recommendations for 
improvement. 
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IV. RESULTS AND DATA ANALYSIS 


A. DATA PREPARATION 

Over 4.5 gigabytes of data was colleeted throughout all three subjeet sessions. 
One challenge was determining how to partition the data such that it could be analyzed 
and presented in a logical and efficient manner. Another was which data to use, 
particularly the eye-tracking output of the EyeWorks software, as there are a number of 
analytic options and some data was not directly relevant to this thesis or its scope. The 
purpose was to provide actionable recommendations to TRAC Monterey to improve the 
usability and workflow of the DaViTo software. As such, not all resultant data was 
analyzed and will not be discussed here. 

The sessions had distinct divisions which made for a natural separation of the data 
into three sections. The purpose of the first interaction with the DaViTo software 
following the tutorials and initially in the beginning of session three was to conduct tasks; 
which became one grouping of information for data analysis. Following task completion, 
the subjects would analyze graphs and answer questions in order to determine the best 
data representation method. This became an additional grouping for analysis. Finally, 
the experiment subjects completed a series of questions at the end of each session. These 
questionnaire answers became the third grouping for analysis. 

B. DAVITO TASK PERFORMANCE AND INTERACTION 

The first two session tutorials concluded with a series of tasks and the final 
session began with similar tasks. Task completion, meaning the user completed the 
directed task, as well as time and correctness, were used to evaluate the ability of the 
participant to utilize the basic skills learned from the tutorials, such as navigation, file 
operations, and creating a data representation. This was not an effort to evaluate 
participants from a training transfer perspective, but rather to determine the efficiency 
and workflow with which simple tasks could be completed by a novice user with minima! 
training on the basic functionality and tools available in DaViTo. Tasks in the first two 
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sessions were used to reinforce the skills taught by the tutorial, and will not be discussed. 
The final session required execution of a series of eight tasks with individual instructions, 
followed by a more complex task divided into three general instructions. Of the 24 
participants, only one achieved a perfect score by completing tasks ten and eleven with 
correct answers. Previous incorrect answers in tasks one through eight would not affect 
completion of tasks nine through eleven, however failure to properly open the data set 
and Shapefile in task nine would prohibit completion of task ten. No partial credit was 
given. 

Three separate analyses were conducted on the session three task data: number of 
mouse clicks for each task, duration of time to complete a task, and cumulative heat 
mapping of the screen area for each task. Mouse click data and time duration of task 
completion were analyzed to determine efficiency of operation with the interface. 
Cumulative gaze density mapping was analyzed for workflow processes, showing gaze 
concentration areas in a heat map to determine if subjects were looking at the appropriate 
part of the screen for completing a task. 

1. Analysis of Mouse Clicks, Correctness, and Completeness of Final 
Session Tasks 

Mouse clicks for each participant were recorded automatically by the EyeWorks 
software. This was compared to a benchmark number of mouse clicks established by the 
experiment designers as the most efficient possible manner to complete the tasks 
correctly. Task descriptions for session three are in Appendix E. 

The summary results for the tasks in session three are displayed in Table 9. This 
is also graphically represented in Eigures 3 and 4. The most notable result from the 
mouse click analysis was the large deviation from the baseline for tasks four and ten. 
Task four directed the participants to “create a rectangle chart on the data point in the 
southwestern most part of Kenya and display the chart”. As seen in Table 9 the 
benchmark for this task was 10 mouse clicks, yet the average number of clicks for 
subjects was 25.43, with a standard deviation of 12.47. This difference from benchmark 
to subject average is displayed in Eigure 3. While users who completed the task correctly 
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improved to 22.63 clicks, the average was far worse for users that could not complete the 
task correctly, with 31.86 clicks, as shown in Figure 4. Task ten was a more complex 
task, directing the subjects to “please answer the following question: what were the total 
events for lED attacks for 2010 in the most northwestern provincial boundary”. Table 9 
shows that on average subjects clicked 15.87 times to complete the task, while the 
benchmark was 5 clicks, with a standard deviation of 12.68. This deviation between the 
benchmark and subject average is shown in Figure 3. Figure 4 shows that for subjects 
who did not complete the task correctly they averaged only 15 clicks, while those who 
did averaged 35 clicks. This indicates that some subjects gave up on task ten. Both tasks 
required the subjects to make a shape within which a graph would be made. Once the 
shape is made the user must click outside of the shape to get the graph to populate within 
it. Frequently when doing this the program would lag, and without a progress status, 
users would be repeatedly clicking outside the shape in an effort to get the graph to 
populate. Some participants even gave up waiting and moved on to the next task, 
particularly for task ten. This is indicative of a usability deficiency within the software. 


Task 

Number 

Bench 

Mark 

Average 
Mouse Clicks 
for All Users 

Average Mouse 
Clicks for 
Correct Answers 

Average 
Mouse Clicks 
for Incorrect 
Answers 

Min 

Max 

Std 

Dev 

Taskl 

4 

6.52 

6.52 

0 

5 

19 

3.31 

Task 2 

3 

7.00 

7.00 

0 

4 

36 

8.62 

Task 3 

3 

3.48 

3.48 

0 

1 

8 

1.27 

Task 4 

10 

25.43 

22.63 

31.86 

7 

53 

12.47 

Task 5 

3 

4.13 

4.33 

2 

1 

15 

2.96 

Task 6 

3 

5.30 

5.32 

5 

3 

22 

4.17 

Task? 

3 

6.04 

6.04 

0 

3 

31 

6.59 

Task 8 

4 

5.70 

5.50 

5.91 

3 

15 

3.43 

Task 9 

14 

19.09 

18.35 

24 

12 

42 

7.88 

Task 10 

5 

15.87 

35.00 

15 

1 

45 

12.68 

Task 11 

2 

3.13 

2.42 

6.5 

1 

23 

4.68 


Table 9. Summary of mouse click and correctness statistics for all session three 
participants. 
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Figure 3. Graph comparing average participant mouse clicks to the bench mark. 


Mouse Clicks for Complete and Incomplete Tasks 



♦ Average Mouse Click 
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Figure 4. Graph comparing the bench mark to average participant mouse clicks for 
both complete and incomplete session three tasks. 


2. Analysis of Time Duration, Correctness, and Completeness of Final 
Session Tasks 

The time for every participant to complete each session three task was 
automatically recorded by the EyeWorks software. This was then compared to a 
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benchmark time established by the experiment designers as the most efficient possible 
time in which to complete the tasks correctly. Task descriptions for session three are in 
Appendix E. 

The summary results for the duration of tasks in session three, and percent correct 
are displayed in Table 10. The most notable result from the analysis of time to complete 
tasks, and similar to the mouse click analysis, was the large deviation from the baseline 
for tasks four and ten, shown in Figure 5. Again, task four directed the participants to 
“create a rectangle chart on the data point in the southwestern most part of Kenya and 
display the chart.” While the bench mark for task four was 62.09 seconds, on average it 
took subjects more than 100 seconds longer to complete it, with a standard deviation of 
67.02, shown in Table 10. There was little deviation between users who were able to 
correctly complete task four from those who could not, but the large deviation for both is 
shown in Figure 6. Task ten was a more complex task, directing the subjects to “please 
answer the following question: what were the total events for lED attacks for 2010 in the 
most Northwestern provincial boundary.” Similar to the mouse click data. Table 10 
shows a large deviation from the baseline of 63.36 seconds for this task. The average 
user completed this task in 141.47 seconds with a standard deviation of 76.17. This time 
was even longer for subjects who completed the task correctly, with Figure 6 showing the 
deviation between the baseline, incorrect, and correct task completion. Note that task 
completion refers to finishing the task, not answering the question correctly. Only one 
subject was able to answer the question asked in task ten correctly. Both tasks required 
the subjects to make a shape within which a graph would be made. Once made, the user 
must click outside of the shape to get the graph to populate within it. DaViTo would 
frequently lag when doing this, and without a progress status, users would repeatedly 
click outside the shape in an effort to get the graph to populate. Confusion from the delay 
and no status indication caused some subjects to give up, having no idea if the software 
was processing their command, and is indicative of a usability deficiency. 
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Session 3 
Task 

Bench 

Mark 

Average Time Per 
Task (sec) 

Min 

Max 

Std Dev 

Percent 

Correct 

Taskl 

10.72 

25.38 

9.50 

78.13 

17.17 

100 

Task 2 

42.96 

52.66 

10.53 

202.92 

36.46 

100 

Task 3 

9.55 

14.86 

3.25 

48.66 

9.27 

100 

Task 4 

62.09 

165.69 

52.28 

319.81 

67.02 

70 

Task 5 

8.87 

17.34 

2.30 

66.99 

15.12 

91 

Task 6 

6.77 

31.18 

8.68 

96.76 

21.27 

96 

Task? 

8.62 

28.34 

9.02 

140.33 

34.70 

100 

Task 8 

11.75 

33.77 

9.23 

149.94 

35.50 

52 

Task 9 

90.93 

90.61 

24.18 

245.10 

46.46 

87 

Task 10 

63.36 

141.47 

7.17 

355.25 

76.17 

4 

Task 11 

16.63 

16.72 

4.45 

281.58 

57.54 

83 


Table 10. Summary of time duration statistics for all session three participants. 
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Figure 5. Graph comparing average participant task durations to the bench mark. 
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Task Duration for Complete and IncompleteTasks 
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■ Completed Tasks 
▲ Incompleted Task 


Figure 6. Graph comparing the bench mark to average participant time duration for 
both complete and incomplete session three tasks. 


3. Analysis of Cumulative Gaze Frequencies of Final Session Tasks 

Along with time duration and number of mouse click to complete each task, the 
EyeWorks Analyze software suite has the capability to generate heat maps and associated 
graphs. The heat maps were useful to display the gaze concentration of each participant, 
or cumulatively for each task. The gaze observation bar graphs were then used to 
compare gaze concentration in the designated regions of the user interface, similar to 
those represented in Figure 2. Task descriptions for session three are in Appendix E. 
Heat maps and gaze concentration graphs for all eleven tasks conducted in session three 
are in Appendix B. Note that on the heat map, color increasing from blue to yellow and 
then to red indicates a greater number of fixations. Also, there are two numbers 
displayed in each region on the heat map. The top number correlates to the percentage of 
total time the participants looked in that region. The bottom number is the percentage of 
total clicks that were made by subjects in that region. 
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Following analysis of the heat maps and gaze observation graphs for each task, 
three of the maps were found to be of specific interest. In addition to reinforcing the 
conclusions reached from the mouse click and time duration data with regards to tasks 
four and ten, an additional deficiency was found in task one. 

Task one was a simple task that had been completed in sessions one and two with 
the tutorial. The user was instructed to open the file WITS.csv in order to populate the 
Map and Data Display Area with a dataset. Figures 7 and 8 show the gaze data for task 
one. This required the user to open the file using the Data Set region of the interface. 
Note in Figure 8 that while this was the predominant region where gazes were observed, 
a significant number were also recorded in the Map area as well as the Map Controls 
area. Figure 7 also displays a large concentration of users gazing at the Map Controls 
area, and the mouse icons representing mouse clicks there. It is expected that there are a 
large number of gazes in the Map area due to its size and the fact that the user would 
focus or scan there while waiting for the dataset to populate. However, the mouse clicks 
and concentration of gaze in the Map Controls area indicates that participants showed a 
tendency to go to this region of the interface when trying to open a file. This is consistent 
with other software programs that have a menu ribbon at the top of the page with 
functionality to open a file, but there is no functionality such as this in DaViTo. This 
again indicates a usability deficiency of the DaViTo software. 

Finally, the conclusions reached from the task duration and mouse clicks data is 
further reinforced by the heat maps for task four, shown in Figure 9, and task ten, shown 
in Figure 10. Tasks four and ten required subjects to create a user defined polygon, and 
allow it to populate with a graph of activity within the polygon. These heat maps show 
the large number of mouse clicks during these tasks, as indicated by the mouse icons in 
the heat maps. This illustrates the unnecessary mouse clicking that was done during 
these tasks. Additionally, these tasks have significant gaze concentrations in the Map 
Controls area, indicating that users may be searching there for the shape drawing 
functionality that requires right clicking in the Map and Data Display area. This further 
supports the usability deficiencies previously discussed in this chapter with regard to 
excessive mouse clicking and missing functionality in the Map Controls ribbon. 
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Figure 7. Final session task one heat map and mouse elieks. Task one direeted subjeets to open the WITS.esv data file, whieh 
required interaetion with the Data Sets region. 
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Figure 8. Graph of final session, task one, eumulative eounts of gaze observations for 
all subjeets. Counts correlate to the number of subject fixations in each 
region shown in Figure 7. 
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Figure 9. Final session task four heat map and mouse clicks. Task four directed subjects to create a user defined polygon and 
display the chart. 
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Figure 10. Final session task ten heat map and mouse clicks. Task ten directed subjects to answer a question that required the 
creation of a user defined polygon, and to display the chart. 
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4. 


Task Results Summary 


Two usability deficiencies were identified by the output data from the session 
three tasks. The first was identified by the excessive mouse clicks when attempting to 
generate a graph within a user-defined polygon. This deficiency is related to the attribute 
of efficiency. The quickness with which a user can complete their goal is reduced when 
time is wasted awaiting program response. Since the product does not react in the 
expected manner, effectiveness is also reduced. The survey results will also show that 
these issues decreased the attribute of user satisfaction. The second deficiency was 
identified by gaze concentrations indicated on heat maps in the improper areas. This 
indicates a lack of functionality that the user expects in the header menu bar ribbon, and 
is indicative of reduced effectiveness of the DaViTo software. Recommendations for 
improvement are discussed in Chapter V, and a consolidated list in Appendix A. 

C. GRAPH PERFORMANCE RESULTS 

A series of data representation questions followed the task completion sections for 
all three sessions. This portion of the experiment was used to evaluate the ability of the 
participant to interpret graphical output similar to that already used in DaViTo, or able to 
be used. The purpose was to ensure that data is being represented in the best manner for 
the user to consistently interpret it quickly and accurately. It should be noted that after 
session one, a brief training session was given by the experiment administrators, followed 
by a review of their performance. The purpose was to see how a user would interpret the 
graphs with no training on their format. Again, this was not an effort to evaluate 
participants from a training transfer perspective, but rather an attempt to determine which 
graphs best represented the type of data being used in DaViTo. It is recognized that a 
great deal of research has been done on data representation methods and there are novel 
concepts in literature that could be applied to future applications; however, this was an 
effort to compare the currently used methods with some others that are easily achievable 
with the current software. Complete statistical analysis of all graph data is contained in 
Appendix C. 
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JFreeChart and R have the capability to present data in a number of different 
graphical formats. These different types are accessible within DaViTo by an advanced 
user; however, the novice user has two readily accessible graph style options. The 
default option is the line chart format (LCF), shown in Figure 11. The LCF has options 
for line chart shapes or markers on (default) or off, as well as the ability to manipulate the 
line thickness. The other option is to use the cumulative stacked bar format (SBF) shown 
in Figure 12. These options are selectable in the display region, shown in Figure 2, of the 
DaViTo interface. Additionally, data can be exported from DaViTo for analysis with 
other statistical tools such as Excel or JMP. 



Figure 11. Example line chart format (ECE) with markers from the DaViTo software. 
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Figure 12. Example cumulative stacked bar format (SBF) from the DaViTo software. 

Five different graph types were constructed for the data representation portion of 
the experiment. Each format is summarized in Table 11. In the experiment sessions 
there were questions about each graph format, for which the data was randomly generated 
from session to session. These questions are listed in Table 12 with a letter designator. 
This allowed the use of the same questions for each graph, but each had a different 
answer. These graph types were chosen because two of them are already available for 
selection by the novice user, and DaViTo has the capability to easily generate the others. 
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Type and 
Abbreviation 


Description 


Example 


Line Chart 
Format 
(LCF) 


Standard line chart, 
no markers for 
inflection points or 
delineators. 



Stacked Line 
Marker 
Format 
(SLMF) 


Line chart with 
markers for 
delineators, and 
cumulative in the y 
direction. 



Event D 
Event C 
Event B 


Event A 


Stacked 
Area Format 
(SAF) 


Filled area chart, 
no markers for 
inflection points or 
delineators, and 
cumulative in the y 
direction. 



■ Event D 

■ EventC 

■ Event B 

■ Event A 
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Type and 
Abbreviation 


Description 


45 

40 


Example 


+ 


Stacked Bar 

Cumulative stacked 

Format 

bar chart. 

(SBF) 



(N<NfSrMrN'NiNl^(NlN(N<NCnroPfim(n 



■ Event D 

■ EventC 

■ Events 

■ Event A 


Clustered 

Columns 

Format 

(CCF) 


Columns for each 
data point are 
clustered side by 
side. 



■ Event A 

■ Event t 
m EventC 

■ Event C' 


Table 11. Table showing the following information for the five graph types constructed 
for the experiment: Type and abbreviation, description, and an example of 
each. 


Question 

Designation 

Session 

Question Text 

A 

One 

What were the total events for December 2012? 

B 

One 

What event had the highest frequency in Jun 2013? 

C 

One 

What month had the most events? 

D 

Two 

What month had the least events? 

E 

Two 

Which month had the highest frequency for event D? 

F 

Three 

Compare the months of December 2012 and December 2013. 
What was the increase or decrease in total events? 

G 

Three 

What event had the highest sum for the months of February 
2012 and February 2013? 


Table 12. List of questions for each session and their designation letter. 
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1. 


Analysis of Session One Graph Data 


The participants were introduced to the five graph types identified in Table 11 in 
session one. Three identical questions were asked about each graph type, listed in Table 
12. These questions were similar to those a possible user in the field or analyst would 
ask, such as how many events occurred in this region in a certain month this year, when 
attempting to conduct trend analysis of a large array of data. Subject answers were 
evaluated for time to answer and accuracy of their answer. Several subjects exhibited 
frustration with some of the graph types due to the amount of data contained or the 
format in which it was displayed. Upon completion of session one each graph type was 
explained to the subjects, and their performance reviewed. This was to determine if 
another graph type would perform better with a simple explanation, such as one that 
could be stated as a mouse-over instruction programmed into DaViTo. 

The stacked bar format (SBF) performed well in session one, with 80.56 percent 
correct answers and an average time to answer of 27.8 seconds, as shown in Table 13. 
The SBF outperformed all other chart types by more than 25 percent in percent correct, 
and was second in time to answer to the stacked area format (SAF) by less than three 
seconds. Despite the slightly slower time to answer, the much larger difference in correct 
answers makes this the better performing format. Statistics for all five chart types are 
shown in Table 13. Figure 13 shows a comparison of the data with top performance in 
the upper right corner. This figure highlights the large difference in performance from 
the other formats, and the small time advantage of SAF. Additionally, subjects answered 
the questions quickly with SBF, which posted the second fastest average time to answer a 
question. It is also of note that the SBF outperformed the other types in every question in 
session one, with these additional statistics in Appendix C. 

The distant second place format in accuracy was the clustered columns format 
(CCF), although it took the longest for users to evaluate. While it had much worse 
performance on average, it should be noted that on question B in Table 12, its 
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performance was similar to that of the SBF, with additional statistics in Appendix C. The 
CCF and SAF graphs had similar accuracy performance to the LCF. The worst 
performance in accuracy was the SLMF. 


Session 1 

Accuracy (% Correct) 

Time (sec) 

CCF 

54.17 

58.33 

SBF 

80.56 

27.81 

LCF 

47.22 

38.38 

SLMF 

22.22 

33.52 

SAF 

51.39 

24.86 


Table 13. Overall statistics for all graph types in session one. 
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Figure 13. Plot of session one time to answer versus accuracy of answer. Note the 
decreasing time on the Y-axis. Top performance is in upper right comer. 

2. Analysis of Session Two Graph Data 

In session two the participants were asked the same series of questions, listed in 
Table 12, about each of the five graph types identified in Table 11. Two identical 
questions were asked about each graph type, similar in type to those of session one. 
Subject answers were evaluated for time to answer and accuracy of their answer. Again, 
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several subjects exhibited frustration with some of the graph types due to the amount of 
data contained or the format in which it was displayed. Upon completion of session two 
there was no review of performance or explanation of graph formatting. 

In session two all chart types showed improved accuracy performance except the 
SBF, which exhibited a significant decrease, as seen in Table 14, although it had the 
shortest time to answer. Figure 14 shows that all types except the LCF were clustered 
near the upper right comer, with the CCF resulting in the highest number of accurate 
answers in commensurate time with the less accurate SBF. 

The format with the second best accuracy was the SAF with 75 percent accuracy, 
and approximately five seconds more required to answer than the CCF. While on 
average it had worse performance than the CCF, it should be noted that on question D, 
from Table 12, it had the highest performance and lowest time to answer. Converse to 
this, on question E in Table 12 it had the lowest performance, with the additional 
statistics in Appendix C. 


Session 2 

Accuracy (% Correct) 

Time (sec) 

CCF 

83.33 

30.38 

SBF 

64.58 

28.88 

LCF 

50.00 

42.79 

SLMF 

72.92 

29.31 

SAF 

75.00 

34.62 


Table 14. Overall statistics for all graph types in session two. 
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Figure 14. Plot of session two times to answer versus accuracy of answer. Note the 

decreasing time on the Y-axis. Top performance is in the upper right corner. 


3. Analysis of Session Three Graph Data 

In session three, the participants were again asked a series of questions from 
Table 12 about each of the five graph types identified in Table 11. Two identical 
questions were asked about each graph type, similar in type to those of sessions one and 
two. Subject answers were evaluated for time to answer and accuracy of their answer. 
Once again, several subjects exhibited frustration with some of the graph types due to the 
amount of data contained or the format in which it was displayed. Upon completion of 
session three there was no review of performance or explanation of graph formatting. 

In this session the SBF and CCF were the top performers in accuracy at 75 
percent, although the CCF took on average greater than 15 seconds to answer, as shown 
in Table 15. Figure 15 shows that SBF, CCF, and SAF were clustered near the upper 
right comer with the best results. 
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Session 3 

Accuracy (% Correct) 

Time (sec) 

CCF 

75.00 

54.62 

SBF 

75.00 

39.09 

LCF 

50.00 

87.31 

SLMF 

47.92 

60.16 

SAF 

60.42 

45.69 


Table 15. Overall statistics for all graph types in session three. 
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Figure 15. Plot of session three times to answer versus accuracy of answer. Note the 
decreasing time on the Y-axis. Top performance is in upper right comer. 


4. Analysis of Overall Graph Data 

The results in Table 16 clearly indicate that the SBF was the top performer in both 
accuracy and time, at 73.38 percent and 35.06 seconds, respectively. The SLMF had the 
worst accuracy and the LCF the worst time, also in Table 16. SBF and LCF formats are 
already in use in DaViTo, with the LCF as the default type. Figure 16 shows that CCF 
was a close second in accuracy, but on average takes 50 percent more time to answer the 
question than with the SBF. 
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While the SBF was clearly the top performer, there are other things to note when 
looking at the overall performance. If only one type of graph were to be used in DaViTo, 
the SBF would be the recommended one; however, other formats have strong points as 
well and should be utilized. Figure 17 shows the performance of all formats for each 
question. Six of the seven questions that used the SBF graph are clustered in the upper 
right hand corner showing consistent top performance. The inconsistent outlier for the 
SBF was question E at 42 percent, which was one of the higher performing questions for 
the LCF at 75 percent accuracy. The CCF also performed well on question E at 87.5 
percent. This indicates that no one graph type is appropriate for all data representations 
or questions. 


All Sessions 

Accuracy (% correct) 

Time (sec) 

CCF 

70.83 

47.78 

SBF 

73.38 

31.93 

LCF 

49.07 

56.16 

SLMF 

47.69 

40.99 

SAF 

62.27 

35.06 


Table 16. Overall statistics for all graph types in all sessions. 


All Sessions Time vs Accuracy 


Accuracy (% correct) 
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Eigure 16. Plot of all three sessions times to answer versus accuracy of answer. Note the 
decreasing time on the Y-axis. Top performance is in upper right comer. 
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Figure 17. Plot of all questions times to answer versus accuracy of answer. The labels 
are the graph type followed by the question identifier. Note the decreasing 
time on the Y-axis. Top performance is in upper right comer 

5. Graph Results Summary 

Two usability deficiencies were noted as a result of the data representation portion 
of the experiment. Recommendations are consolidated in Chapter V. 

The first identified deficiency for data representation is related to the attribute of 
usefulness, and was the inability to answer all questions with considerably high 
performance with the default graph format used by DaViTo. The default LCF graph is 
the lowest performing graph of the five tested. Fortunately, the highest performing 
graph, the SBF, is also readily available for use with the ease of a simple radio button in 
the Display region of the DaViTo interface. Using the results of both of these graph 
formats all questions were able to be answered with an accuracy that was on average in 
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excess of 70 percent. However, this puts the user in the position of recognizing that a 
different graph would relay information better, and have the technical prowess to change 
the graph type. This also results in deficiencies in the attributes of efficiency, 
effectiveness, and likely satisfaction. 

The second identified deficiency is in the attributes of efficiency and 
effectiveness. As seen in Figure 17, different graphs perform better for different types of 
questions, as well as different users. While the SBF chart performed well on six out of 
seven questions, there were other formats that performed better on certain questions. For 
example, the SBF performed well at 75 percent accuracy on question G; however it was 
outperformed by CCF, which had 100 percent accuracy. In question by question 
comparison, one can see which types perform better and that a cumulative type graph 
such as SBF is not always better for questions that involve single event types rather than 
totals, as no mental math is required to derive the number from the stack of bars. The 
results of individual questions can be seen in Appendix C. 

The worst performing graph types were the LCF and SLMF. The LCF is the 
default choice in DaViTo, and can also be used with markers like the SLMF. While they 
had overall accuracy averages of less than 50 percent and some of the longest average 
times, they performed well at some questions. This is again evidence that having 
multiple options gives users the capability to choose the type that best conveys the data 
they are trying to represent. 

D. SURVEY RESULTS 

Each session was concluded with a survey to allow the participant to give 
feedback about the training and software. For sessions one and two, there were three 
common questions, which were also the first three questions for the final session. The 
final session concluded with a more extensive survey with questions related to the five 
usability attributes listed in Table 1, as well as categories of improvements and 
visualizations. All questions are contained in Appendix F. Unless otherwise noted, 
questions that were not free-form text entry by the subject used the number key shown in 
Table 17. 
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Strongly 

Disagree 

Disagree 

Neither Agree 

nor Disagree 

Agree 

Strongly Agree 

1 

2 

3 

4 

5 


Table 17. Key used by subjects to answer survey questions that were not free-form text. 


1. Analysis of the Three Common Questions from Each Session 

Three standard questions were used for the first two sessions and were contained 
in the final session questionnaire. These questions were standardized across each session 
to allow for comparison. The only exception is question one for session three, as it 
included no training. 

The first common question asked the subjects to evaluate the training during that 
session. The session three question was not evaluated against sessions one and two 
because there was no training, and will not be discussed. The training was evaluated on a 
five point scale from very good to very poor, similar to Table 17. The results of common 
question one for sessions one and two are shown in Table 18. This high valuation may 
give TRAC Monterey guidance on the development of future training tools to distribute 
with the software. Full analysis can be found in Appendix D. 


Session 

Average Response for Question 1: How 
would you rate the training session? 

1 

4.6 

2 

4.5 


Table 18. Common question one results. 


The second common question required the subjects to self-evaluate the amount of 
additional training they may need in addition to that already received to feel comfortable 
using DaViTo in the field. This question was evaluated on the scale shown in Table 19. 
Figure 18 shows that subject confidence decreased between sessions one to two as 

subjects felt they needed 14 percent more training. Following the final session, 
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participant’s confidence inereased and the amount of additional training thought to be 
needed deereased by 21 pereent to its lowest level. This translates to a total of five to 
seven hours of training are thought to be required in addition to the two 20 minute tutorial 
sessions. This response may give TRAC Monterey guidanee on the development of 
future training tools and the amount of additional training needed. For full analysis 
results see Appendix D. 


0-1 Hours 

2-4 Hours 

5-7 Hours 

8-10 Hours 

10 + Hours 

1 

2 

3 

4 

5 


Table 19. Evaluation seale for eommon question two. This question asked subjeets: 

“how many more hours of training do you think would be neeessary before 
you are ready to use DaViTo in the field for analysis?” 



Figure 18. Graph of average response to eommon question two. Note that the y-axis 
eorrelates to the scale in Table 19. 


The final eommon question asked the subjeets to list their likes and dislikes about 
the DaViTo program for that session. Unlike the final survey where questions were 
written with speeifie regard to the usability attributes deseribed in Table 1, these 
questions have no eontent to lead the subjeet and are strietly free-form entry. The results 
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of common question three are shown in Tables 20, 21, and 22 for sessions one, two, and 
three, respectively. Full transcription of comments can be found in Appendix D. 
Common liked features across all three sessions were ease of use and the user interface 
layout. Navigation functionality, simplicity, and geo-data representation appeared in the 
top five in more than one session. The single common disliked feature across all three 
sessions was response time. Mouse interaction, lack of a progress status notification, and 
graph functionality were listed in the top five on more than one occasion. It is also of 
note that some features were liked by some participants while disliked by others. 
Recommendations for improvement based on the liked and disliked features are 
discussed in Chapter V. 


Top 5 Liked Features 

Times 

Cited 

Top 5 Disliked Features 

Times 

Cited 

Ease of Use 

24 

Response Time 

20 

Layout 

14 

Distance Function 

10 

Navigation Functionality 

8 

Layout 

6 

Simplicity 

7 

Map Overlays 

6 

Data Sets 

6 

Navigation Function 

4 


Table 20. Summary statistics of the top five likes and dislikes for session one, common 
question three. 


Top 5 Liked Features 

Times 

Cited 

Top 5 Disliked Features 

Times 

Cited 

Geo-Data Representation 

6 

Mouse Interaction 

7 

Ease of Use 

5 

Response Time 

7 

Functionality 

3 

No Progress Status 

3 

Layout 

2 

Creating a Graph 

2 

Graph Functionality 

1 

Graph Functionality 

2 


Table 21. Summary statistics of the top five likes and dislikes for session two, common 
question three. 
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Top 5 Liked Features 

Times 

Cited 

Top 5 Disliked Features 

Times 

Cited 

Ease of Use 

4 

Response Time 

14 

Geo-Data Representation 

3 

No Progress Status 

9 

Navigation Functionality 

2 

Graph Functionality 

9 

Layout 

2 

Map Labels 

3 

Simplicity 

1 

Mouse Interaction 

3 


Table 22. Summary statistics of the top five likes and dislikes for session three, 
common question three. 


2. Analysis of the Final Survey 

The final survey was written utilizing the five usability attributes described in 
Table 1, from the Handbook of Usability Testing (Rubin & Chisnell, 2008). Topics of 
improvements and visualization were also used to formulate questions. All questions, 
analysis, and responses are contained in Appendix D. 

1. Usefulness 

The attribute of usefulness refers to the ability of a user to achieve their 
goals and their willingness to use the product. If this attribute is not considered in the 
development process designers may create software from their point of view rather than 
that of the user, resulting in a deficient design (Rubin & Chisnell, 2008). 

Questions concerning usefulness, shown in Table 23, all had a score 
greater than three except Use_6, which scored a 2.66. It is of note that the user’s manual 
was not consulted by any subject despite being available, and there is no electronic user’s 
manual or help function accessible from the DaViTo interface. While the majority of 
subjects selected response three (neither agree nor disagree), the other users based their 
response on the perceived difficulty of the software rather than by actually referencing 
the manual. This indicates a potential deficiency in usefulness with regards to the ability 
to get assistance from a manual or help tab within the interface. Question Use_2 results 
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also identify usefulness deficiencies in the areas of graph interpretation and construction 
as well as polygon construction. Deficiencies will be addressed by recommendations in 
Chapter V. 


Usefulness Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 



1 

1 




The tasks were easy 
to complete? 


2 

1 




Use_l 

3 

5 

3.71 

4 

4 


4 

14 






5 

3 




The terminology for 
buttons and 
commands was clear 
and understandable? 


1 

0 





2 

4 




Use_3 

3 

2 

3.79 

4 

4 


4 

13 





5 

5 




A computer novice 
would be able to use 


1 

1 





2 

7 




this software with 

Use_4 

3 

4 

3.16 

4 

3 

the training 
provided? 


4 

11 





5 

1 




A computer novice 
would be able to use 


1 

4 





2 

5 




this software with 

Use_6 

3 

11 

2.66 

3 

3 

just the user’s 


4 

3 




manual? 


5 

1 




What was the most 


Reading/interpreting the graphs. 

difficult task to 

Use_2 

Creating the graphs. 

complete? 


Drawing polygons. 


Table 23. Table of usefulness questions and associated statistics. 
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2, Effectiveness 


The attribute of effectiveness refers to whether the product behaves as the 
user expects or that users can use the software to do what they desire it to do. While this 
can be measured against a baseline rate of error, these questions addressed the users’ 
perceived effectiveness of their interaction with DaViTo (Rubin & Chisnell, 2008). 

Questions concerning effectiveness, listed in Table 24, all had a score of 
three or greater except Effect_4, which scored a 1.875. The results of this question once 
again address the latency of the program and point to an effectiveness deficiency. This 
can be a source of user frustration, also affecting usefulness if users are driven to stop 
using the software. Question Effect_3 also identifies effectiveness deficiencies in the 
areas of software lag, the lack of a processing status bar, and the need for geographic 
labels within the Map and Data Display region. Deficiencies will be addressed by 
recommendations in Chapter V. 
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Effectiveness 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

I understood where I 


1 

1 




was at in the 


2 

6 




software as I 

Effect_l 

3 

3 

3.375 

4 

4 

progressed through 


4 

11 




a task? 


5 

3 




The mouse 
interacted with the 
software as I 
expected it to? 


1 

1 





2 

4 




Effect_2 

3 

2 

3.667 

4 

4 


4 

12 





5 

5 






1 

11 




I was satisfied with 


2 

9 




the reaction time of 

Effect_4 

3 

1 

1.875 

1 

2 

the software? 


4 

2 






5 

1 




I felt there was an 


1 

3 




unnecessary amount 


2 

8 




of mouse clicking to 

Effect_5 

3 

7 

3.25 

4 

3 

complete the given 


4 

4 




task? 


5 

2 




I made frequent 
errors trying to 
complete the given 
task? 


1 

2 





2 

9 




Effect_6 

3 

5 

3 

4 

3 


4 

3 





5 

5 




Was the software 


Processing lag or crash. 

simple to use? 

Effect_3 

Eack of a processing status bar. 

Please explain. 


Eack of geographic labels. 


Table 24. Table of effectiveness questions and associated statistics. 
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3. Efficiency 


The attribute of efficiency concerns the speed with which the user can 
accomplish a goal(s). While usually measured against a baseline time, these questions 
address the user’s perceived effective interaction with DaViTo (Rubin & Chisnell, 2008). 

Questions concerning efficiency, listed in Table 25, all had a score of three 
or greater except Eff_8 which scored a 2.25. Eff_8 directly pinpoints a serious lack of 
efficiency experienced by multiple users due to software freezes and subsequent crashes. 
This can also be a source of user frustration affecting usefulness if users are driven to 
stop using the software. Question Eff_4 results also identify efficiency failures in visual 
representation as a result of no geographic labeling of countries or regions, and the 
inability within DaViTo to sort different data types within a data set and display their 
data points with different colors. This question also had multiple responses again 
identifying difficulty reading graphs in DaViTo. Deficiencies will be addressed by 
recommendations in Chapter V. 
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Efficiency Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

The organization of 
the screen matched 
my expectation of 
button location? 


1 

1 





2 

4 




Eff_l 

3 

5 

3.5 

4 

4 


4 

10 





5 

4 






1 

0 




Similar icons and 


2 

3 




activities are 

Eff_2 

3 

3 

3.83 

4 

4 

consistently placed? 


4 

13 






5 

5 






1 

0 




Navigation of 


2 

5 




DaViTo was 

Eff_3 

3 

3 

3.58 

4 

4 

intuitive? 


4 

13 






5 

3 






1 

0 




The color scheme for 


2 

3 




the software was 

Eff_5 

3 

4 

3.79 

4 

4 

appropriate? 


4 

12 






5 

5 




The text within 


1 

0 




output graphical 


2 

8 




representations was 

Eff_6 

3 

6 

3.21 

2 

3 

easy to read and 


4 

7 




match to data? 


5 

3 




The software header 
bars and dropdown 
lists made 
navigational sense? 


1 

2 





2 

0 




Eff_7 

3 

2 

3.83 

4 

4 


4 

16 





5 

4 






1 

9 




The software was 
stable - no freezing? 


2 

6 




Eff_8 

3 

3 

2.25 

1 

2 


4 

6 






5 

0 




Was the visual 


Eack of geographic labels. 

representation easy 

Eff_4 

Charts difficult to read. 

to understand? 


Unable to sort data types within a set by color. 


Table 25. Table of effieieney questions and assoeiated statisties. 
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4. Learnability 


The attribute of learnability is a sub-attribute of effectiveness and concerns 
the user’s ability to operate the system, whether it is after some period of familiarization 
and/or training. Additionally, this element can refer to a subject’s ability to use the 
software effectively after some period of inactivity (Rubin & Chisnell, 2008). Both of 
these were addressed by the learnability segment of the final questionairre. 

Questions concerning effectiveness all had a score of three or greater 
except Leam_3, with a score of 2.17, and are presented in Table 26. Similar to the 
second common question of all three sessions, Learn_3 requires the participant to 
evaluate if they are adequately trained to use the software. Like common question two, 
the majority of subjects do think additional training on DaViTo is necessary. Also of 
note is Learn_2, which like Use_6, concerns the use of the user’s manual. The responses 
to Learn_2 averaged three, and 10 subjects agreed or strongly agreed, while Use_6 only 
scored a 2.66 and only four subjects agreed or strongly agreed. Despite these conflicting 
responses the fact remains that the user’s manual was never consulted by any 
participants, and there is little help available via the interface. These deficiencies will be 
addressed by recommendations in Chapter V. 
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Learnability 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

I think most people 
could figure out how 
to use this software 
with the training 
aids provided? 

Learn_l 

1 

3 

3.54 

4 

4 

2 

3 

3 

0 

4 

14 

5 

4 

I think most people 
could figure out how 
to use this software 
with just the user’s 
manual? 

Learn_2 

1 

3 

3 

4 

3 

2 

5 

3 

6 

4 

9 

5 

1 

No more training is 
needed to effectively 
use this software? 

Learn_3 

1 

6 

2.167 

2 

2 

2 

11 

3 

4 

4 

3 

5 

0 

If given these tasks 
in one month to 
complete again-1 
feel I could execute 
them? 

Learn_4 

1 

5 

3.22 

4 

4 

2 

2 

3 

3 

4 

10 

5 

4 

I could show 
someone who has no 
experience with 
DaViTo basic 
functionality? 

Learn_5 

1 

1 

3.79 

4 

4 

2 

2 

3 

2 

4 

15 

5 

4 

More training aids 
are needed for me to 
effectively learn to 
use the software? 

Learn_6 

1 

0 

3.5 

4 

4 

2 

6 

3 

4 

4 

10 

5 

4 


Table 26. Table of learnability questions and associated statistics. 
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5. Satisfaction 


The attribute of satisfaction is linked to the users’ perception of the 
software. Typically users will perform better when using a satisfying product that meets 
their needs (Rubin & Chisnell, 2008). 

Questions concerning satisfaction, listed in Table 27 had three results with 
scores less than three. These questions are all related to the participants self-assessment 
of their abilities to use DaViTo for basic analysis (Sat_l), complex tasks without the 
user’s manual (Sat_3), and using DaViTo with no assistance (Sat_6.) Overall, this is 
indicative of a deficiency in satisfaction with their ability to use the software and is likely 
related to their current level of training and experience. Again, the user’s manual was 
never consulted by any participants, and there is no help available via the interface. This 
deficiency will be addressed by recommendations in Chapter V. 
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Satisfaction 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

I feel confident in my 
abilities to use 
DaViTo for basic 
analysis? 


1 

3 





2 

3 




Sat_l 

3 

10 

2.958 

3 

3 


4 

8 





5 

0 




I think 1 can 
successfully finish 
basic tasks without 
the user's manual? 


1 

3 





2 

4 




Sat_2 

3 

6 

3.125 

4 

3 


4 

9 





5 

2 




I think 1 can 
successfully 
complete complex 


1 

8 





2 

6 




Sat_3 

3 

4 

2.375 

1 

2 

tasks without the 
user's manual? 


4 

5 





5 

1 




I think this software 
would add value to 
missions I have 
performed? 


1 

0 





2 

2 




Sat_4 

3 

11 

3.458 

3 

3 


4 

9 





5 

2 






1 

2 




I would recommend 


2 

2 




this software to my 

Sat_5 

3 

12 

3.167 

3 

3 

command? 


4 

6 






5 

2 




I feel comfortable 
using DaViTo 
without any 
assistance? 


1 

3 





2 

8 




Sat_6 

3 

8 

2.67 

3 

3 


4 

4 





5 

1 




Were you satisfied 


Software latency/lag. 

with the 


Lack of help functionality. 

performance of the 
software? Please 
explain. 

Sat_7 

Lack of graphing options and graphs difficult to 
interpret. 


Table 27. Table of satisfaetion questions and assoeiated statisties. 
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6 . 


Visualizations 


The visualization attribute was used to identify subject opinions with 
regard to the visual appearance of DaViTo output and the user interface. These trends 
could then be used to generate recommendations in addition to those generated from the 
five usability attributes. 

Questions concerning visualization are presented in Table 28, and had one 
question with a score below three. This question, Visual_l, had a score of 2.375 and 
asked users if the output data could be clearly interpreted. The large number of negative 
responses correlates with previous questions regarding the quality of data output in 
DaViTo, and reinforces a deficiency in this area. Question Visual_2 results were not new 
comments and are concurrent with similar comments from other areas. These 
deficiencies will be addressed by recommendations in Chapter V. 

Subjects were also questioned on their ability to identify key icons used in 
DaViTo. Five of the six questions were answered with greater than 70 percent accuracy. 
The navigation icon (Visual_15) performed poorly with less than half of users able to 
identify it. While in general icon recognition was not an issue, with the exception of the 
navigation icon, there were multiple comments throughout the free-form portions of the 
survey to remove the requirement to change cursor functionality between the four types 
and instead add right click functionality with options to perform all actions from the 
single cursor type. This will be addressed in Chapter V. 


Visualization 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

The way in which 
output data is 
displayed is clear to 
interpret 


1 

7 





2 

6 




Visual_l 

3 

8 

2.375 

3 

2 


4 

1 





5 

2 




What more would 


More chart options availal 

ble to the user. 

you like to see in the 
output of this 

Visual_2 

Ability to manipulate the charts, such as zoom 
and view data values on the chart. 

software? 


More functionality to tailor the data. 


Table 28. Table of visualization questions and associated statistics. 
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Visualization 

Question 


What does this 
icon mean? 


Question 

Designator 

Icon 

Options Selected 

Frequency 

Visual_13 




Pan 

21 




Map Navigation 

1 

Gestures 

2 




% Correct 

87.5 

Visual_14 


ft 


Distance 

24 

% Correct 

100 

Visual_15 




Navigation 

11 



Pan 

4 

R An 

+1+ 

Map Navigation 

8 



I don’t know 

1 



% Correct 

45.83 

Visual_16 


m 

Forward 

Projection 

21 

Pan 

1 

I don’t know 

2 


% Correct 

87.5 

VisuaLlV 


Map Navigation 

17 


mt 


Navigation 

5 

Pan 

2 


% Correct 

70.83 

Visual_18 


1 


Drawing 

24 

% Correct 

100 


Table 29. Table of visualization questions regarding ieons, and assoeiated statistics. 
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7. Improvements 


The inclusion of the improvements section with the other attributes was an 
effort to give the participants an opportunity to voice ways to change DaViTo in the 
context of specific questions. This prevented generic responses to general questions. 

Questions concerning improvements and some of the results are contained 
in Table 30, with complete data in Appendix D. The question Improve_l asks about the 
user’s manual. While most subjects answered neutrally, there was a full spectrum of 
answers despite the manual never having been consulted. When asked in Improve_9 
about their preferred graphs, the majority of users chose the SBF by a large margin, 
which is concurrent with it exhibiting the top performance overall. Deficiencies 
identified by the subjects in the free-form questions will be addressed by 
recommendations in Chapter V. 
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Improvement 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

User’s manual is 
easily 

understandable? 

Improve_l 

1 

2 

3 

3 

3 

2 

1 

3 

17 

4 

3 

5 

1 

More training aids 
are needed for me to 
effectively learn to 
use the software? 

Improve_2 

1 

0 

3.5 

4 

4 

2 

6 

3 

4 

4 

10 

5 

4 

How would you 
improve DaViTo? 
Please explain. 

Improve_3 

Correct the lag issues. 

Generally, improve the graphs and interaction. 

Add a progress icon. 

What was the most 
frustrating feature of 
DaViTo? 

Improve_4 

Lack of on-screen help available. 

Standardize mouse button functionality. 

General - Lag and charts. 

What did you most 
like about DaViTo? 
Please explain. 

Improve_5 

Ability to display data geographically. 

Integrated data analysis. 

Simple and intuitive interface. 

What would you 
change about the 
color scheme / page 
layout? Please 
explain. 

Improve_6 

Area relief or 3D capability like Google Earth. 

Color scheme is bland, add contrast. 

Change default data point color. 

How would you 
improve the mouse 
interaction with the 
software? Please 
explain. 

Improve_7 

Right click on map shows all available options. 

Quicker response to right clicks. 

Highlighting or positive indication of selection. 

How would you like 
to see the software 
employed? Please 
explain. 

Improve_8 

Multiple, see Appendix D. 

What graph did you 
think conveyed the 
information in 
general the best? 

Imrpove_9 

Options 

Frequency 

SBF 

14 

CCF 

2 

SFMF 

6 

FCF 

0 

SAF 

2 


Table 30. Table of improvement questions and associated statistics. 
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3. Survey Results Summary 


Multiple usability deficiencies were identified by the output data from both the 
three common questions for all sessions, as well as the extensive session three survey. 
Recommendations for improvement will be made in Chapter V. 

The common questions were more general than the session three survey. The first 
two questions did not identify deficiencies, rather were related to the training received 
and subject opinions of additional training necessary. The responses will serve as a basis 
for improvement recommendations. The final question identified general likes and 
dislikes. The dislikes correlate to more specific comments made in the free-form 
questions of the final survey and will be used to formulate recommendations for the next 
release of the DaViTo software. 

The final survey was extensive and exhausting. Multiple deficiencies were 
identified by both free-form response questions and scaled answer questions. These 
deficiencies will be incorporated into the recommendations in Chapter V. 
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V. RECOMMENDATIONS 


The main objective of this thesis was to provide clear and actionable 
recommendations to TRAC Monterey to improve the usability of the DaViTo software. 
Data analysis was divided into three distinct sections for analysis and results reporting, 
and the recommendations will follow this same format. Identical recommendations 
supported with different bases will be reported in the tables. Additionally, 
recommendations contained in the tables in this Chapter are connected to their primary 
usability attribute in Table 1. If other attributes are related, they will be presented in the 
discussion paragraphs. A comprehensive table of recommendations can be found in 
Appendix A. 

A. RECOMMENDATIONS BASED ON TASK RESULTS 

Eye-tracking and interface interaction was the primary source of data for the task 
results analysis presented in Chapter IV. Three data sources were used to generate these 
recommendations: mouse clicks, duration of time to complete tasks, and heat maps of all 
participants gaze locations. 

The first two recommendations listed in Table 31, A.l and A.2, were touched on 

by nearly every subject, and will be mentioned in multiple sections of Chapter V for a 

variety of different bases. Recommendation A1 requires revising the software code to 

improve software processing, response time, and stability. Recommendation A2 is to 

implement a processing status bar to alert the user that the software is operating and is not 

frozen. As discussed in Chapter IV, multiple mouse clicks would take place and wasted 

time elapse as participants waited with growing impatience for the polygon to form and 

populate. This lack of efficiency in the software code resulted in user frustration, which 

the absence of, is one of the definitions of usability mentioned in Chapter II. Not only is 

efficiency at issue here but also usefulness and satisfaction. As DaViTo performance 

deficiencies reduce the ability of the user to complete their goals efficiently, the user then 

becomes less likely to use the product at all, and their perception of the product becomes 

more negative (Rubin & Chisnell, 2008). Finally, most software has lag time at some 
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point as it computes things, and the implementation of A.2 would reduce the 
dissatisfaction of users. As the moments pass at least the user will know the program is 
processing rather than being left guessing as to whether it is simply frozen. 

The excess mouse clicks were the result of frustration and no status of progress; 
however, the mouse clicks should not be necessary at all. The procedure of drawing the 
shape and then right-clicking outside the shape to initiate its formation is unnecessary and 
counter-intuitive. Like other programs with shape drawing functionality (Microsoft 
Paint, PowerPoint, Word, etc.), the code should be modified to not require this additional 
click as recommended by A.3, but rather initiate once the mouse button is released. 

Finally, the heat map analysis indicated a large concentration of subjects gazing 
and clicking in the Map Controls Region. Again, the file manipulations functionality is 
counter-intuitive to common programs used. Recommendation A.4 will increase 
effectiveness for users. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

A.l 

Improve software 
processing, response 
time, and stability. 

Efficiency - When drawing user-defined 
polygons there was excessive lag time or the 
software crashed. 

A.2 

Implement a progress 
status bar. 

Effectiveness - When drawing polygons the lag 
time led subjects to continue clicking outside the 
polygon because there was no indication of 
progress. 

A.3 

Draw and populate a 
polygon when drawn 
by the user. 

Effectiveness - Right clicking outside of the 
polygon was counter-intuitive to users. 

A.4 

Implement file 
operations in the main 
menu ribbon. 

Effectiveness - Heat mapping shows that many 
users looked to the menu ribbon for file 
operations. 


Table 31. List of recommendations and the basis derived from analysis of the task 
results. 
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B. RECOMMENDATIONS BASED ON GRAPH RESULTS 


Three of the four development goals for DaViTo discussed in Chapter II involve 
graphical representation of data. The production of useful visualization and data analysis, 
and representing data with graphical output that can be interpreted by ground 
commanders are the root of two of these (Evangelista, Henry, Buttrey, & Whitaker, 
2012). The usability of DaViTo is significantly deficient in this regard. In addition to the 
two recommendations in Table 32, there are more concerning the functionality of graphs 
in the next section generated from the survey data. 

It was identified that the default graphing option. Line Chart Format (LCF), was 
the lowest scoring option overall in the three sessions. Fortunately the highest performer. 
Stacked Bar Format (SBF), is the one and only other option directly on the DaViTo 
interface. However, this requires more from the user, such as knowing there is another 
option and realizing that this option may better convey the data. Implementing B.l will 
make the SBF the default graph type. This will have an immediate impact on the 
efficiency of the software by giving the user the best chance at successfully interpreting 
the graph. This will also improve the usefulness, effectiveness, and satisfaction attributes 
of DaViTo. 

The second recommendation, B.2, will add more options for different graph types 
and further improve the usefulness of DaViTo. It is clear from the results that while the 
SBF is the top overall performer, different graphs produce better results dependent upon 
the question. Similarly, most software with graphing functionality (Excel, IMP) are not 
limited to two graph types, but give their users a plethora of choices, as well as provide 
brief explanations of the strengths of each format. Admittedly, DaViTo has the 
capability to produce different graph types either through the use of R or importing the 
data to Excel. Unfortunately, this is laborious and counter to one of the goals stated for 
the development of DaViTo which was that the software would be easy to use for 
someone unfamiliar with statistical methods (Evangelista, Henry, Buttrey, & Whitaker, 
2012). Requiring a user that is unfamiliar with statistical methods such as those 
employed by R, or requiring the importation, manipulation, and eventual graphing of data 
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in Excel, is not realistic. Including more options for graph types in the user interface 
such that they are easily accesible will better meet the original development goals of 
DaViTo. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

B.l 

Make the Stacked Bar 
Format (SBF) the 
default graph type. 

Efficiency - The SBF graph was the most 
conducive to generating a correct answer from 
experiment subjects. 

B.2 

Add more options for 
different graph types. 

Usefulness - Different graphs have better 
performance relative to the question asked. 


Table 32. List of recommendations and the basis derived from analysis of the graph 
results. 


C. RECOMMENDATIONS BASED ON SURVEY RESULTS 

Survey results constituted the only source of participant feedback on DaViTo. 
Although subjective, the majority of participants seemed genuinely interested in 
providing useful feedback, ensuring the hours they devoted without compensation to this 
research was not wasted. 

1. Recommendations Based on the Three Common Questions 

Currently, the only training product developed for and distributed with DaViTo is 
a 26 slide PowerPoint presentation. While this covers much of the basic functionality of 
the software, it is not effective at training a user to use the software such that they meet 
the goals stated when developing DaViTo. Implementing recommendations C.l and C.2, 
listed in Table 33, would address the current learnability deficiency in the software. 
Recommendation Cl is to develop a tutorial based training product for DaViTo. While 
the tutorial method developed for this experiment was not itself evaluated against the 
PowerPoint presentation or any other training methods, users did rate these sessions 
highly on exit surveys, with an average of 4.55 on a scale of one to five, and 21 of 24 
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users were able to complete all final session tasks, albeit with only one perfect score. 
Recommendation C2 is to distribute an in depth training product with DaViTo. A 
thoroughly prepared and executed training plan could teach the basic use of DaViTo in 
several hours of training. Final responses after session three indicated that users 
estimated they would require five to seven total hours of training to feel comfortable 
using DaViTo in the field, and 50 percent of users agreed that a novice could use the 
software with just the training provided. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.l 

Develop a tutorial 
based training product 
for DaViTo. 

Leamability - Currently there are no training 
products for DaViTo beyond a basic PowerPoint 
slide show. 

C.2 

Distribute a training 
product with DaViTo. 

Leamability - There is currently a PowerPoint 
slide show distributed with DaViTo. 


Table 33. List of recommendations and the basis derived from analysis of the three 
common question survey results. 


2. Recommendations Based on Survey Usefulness Questions 

Designers did design a Help tab on the DaViTo menu ribbon; however, it 
currently has no functionality beyond providing a link to the OpenMap website help. 
There is no DaViTo specific help and the user’s manual is not available from within the 
software. Linking the user’s manual to the help tab on the ribbon bar to implement 
recommendation C.3 in Table 34, would correct this deficiency. Without even seeing the 
user’s manual, 83 percent of subjects were either neutral or disagreed that a computer 
novice would be able to use DaViTo with only the user’s manual. Despite this, 
accessibility to the user’s manual would increase usefulness by providing a reference for 
assistance when problems arise, and linking it to the help tab would place it where most 
users would expect as it is commensurate with commonly used software. 
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When subjects were asked if the software is simple to use, it was most frequently 
stated that the three most difficult tasks to accomplish involved graph interpretation, 
graph creation, and generating user-defined polygons. Graph tools such as a scrolling 
capability, and mouse-over functionality to display the value of the data point would 
assist in the implementation of recommendation C.4, which is to include graph tools to 
improve graph readability. C.5 recommends making graphs easier to create. This is 
likely related to the excessive mouse clicking required when drawing a polygon and the 
subsequent graph creation. Finally, drawing and populating a polygon when drawn by 
the user, designated recommendation C.6, is identical to A.3, but is based on a usefulness 
deficiency identified by the participants. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.3 

Link the user’s manual 
to the Help tab on the 
ribbon bar. 

Usefulness - Currently the user’s manual is not 
accessible via the DaViTo interface. 

C.4 

Include graph tools to 
improve graph 
readability. 

Usefulness - Graph interpretation was identified 
by subjects as the one of the most difficult tasks. 

C.5 

Make graphs easier to 

create. 

Usefulness - Graph creation was identified by 
subjects as the one of the most difficult tasks. 

C.6 

Draw and populate a 
polygon when drawn 
by the user. 

Usefulness - Polygon functions were identified 
by subjects as one of the most difficult tasks. 


Table 34. List of recommendations and the basis derived from analysis of the 
usefulness portion of the survey results. 
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3. 


Recommendations Based on Effectiveness Questions 


Recommendations C.7 and C.8, are both recommendations to improve software 
processing latency and stability. This recommendation was previously identified with 
recommendation A.3. Similarly recommendation C.9, to implement a progress status bar, 
is identical to A.2. 

The Map and Data Display Area contains legal boundaries for countries, but there 
are no labels. This was identified by subjects as a deficiency when using DaViTo in the 
final survey. Recommendation C.IO in Table 35 addresses this deficiency by 
recommending the implementation of geographic labels in the Map and Data Display 
Area. Specifically, in the final session participants were directed to find Kenya and load 
a Shapefile. This required geographic knowledge of Africa not necessary when using a 
typical map with appropriate labeling. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.7 

Improve the software 
reaction time. 

Effectiveness - 83 percent of subjects were not 
satisfied with the software reaction time. 

C.8 

Improve software 
processing, response 
time, and stability. 

Effectiveness - Subjects stated this as the worst 
deficiency that reduced the simplicity of use of 
DaViTo. 

C.9 

Implement a progress 
status bar. 

Effectiveness - Subjects stated this as the second 
worst deficiency that reduced the simplicity of 
use of DaViTo. 

C.IO 

Include geographic 
labels in the Map and 
Data Display Area. 

Effectiveness - Subjects stated this as the third 
worst deficiency that reduced the simplicity of 
use of DaViTo. 


Table 35. List of recommendations and the basis derived from analysis of the 
effectiveness portion of the survey results. 
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4. 


Recommendations Based on Efficiency Questions 


Recommendation C.ll, in Table 36, is again related to improving software 
processing latency and is the same as recommendation A.3. Similarly, recommendation 
C.12, including geographic labels, is identical to C.IO. 

Recommendation C.13 will allow data sets of different data types to be sorted 
with different colors. Large datasets such as the one used frequently in the experiment, 
WITS. CSV, have multiple data types. Additionally, when multiple datasets are loaded, 
they all continue to use the default gray color scheme. Automatic color coding of the 
different subsets was recommended by multiple subjects to improve the visual 
representation of data on the Map and Data Display Area. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.ll 

Improve software 
processing, response 
time, and stability. 

Efficiency - 67 percent of subjects disagreed that 
the software was stable and not prone to crashing. 

C.12 

Include geographic 
labels in the Map and 
Data Display Area. 

Efficiency - Subjects reported this as the top 
deficiency that made visual representations easy 
to read. 

C.13 

Make data types 
within a data set 
sortable by color. 

Efficiency - Subjects reported this as a deficiency 
that made visual representations difficult to read. 


Table 36. List of recommendations and the basis derived from analysis of the efficiency 
portion of the survey results. 


5. Recommendations based on Learnability Questions 

Most questions in the learnability segment of the final survey reflect positively 
upon the training developed for the experiment. The majority of subjects believe that 
with this training most people could use DaViTo, they could complete the final session 
tasks again in one month, and that they could teach a user the basic functionality of 
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DaViTo. Despite this, a majority also believe more training products are needed. 
Because of these results, Table 37 lists recommendation C.14 (identical to C.l) to 
endorse a tutorial based training program and C.15 (identical to C.2) recommends that it 
is distributed with DaViTo. Training should be developed with the DaViTo software 
objective in mind that a user whom is unfamiliar with statistical methods and Geographic 
Information Systems could use the software. This will reinforce the other software goals 
of users that can produce useful visualizations that can be grasped and understood by 
ground commanders (Evangelista, Henry, Buttrey, & Whitaker, 2012). 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.14 

Develop a tutorial 
based training product 
for DaViTo. 

Leamability - 75 percent of subjects reported that 
most people could figure out how to use DaViTo 
with the tools provided; however 58 percent 
believe more training products are needed. 

C.15 

Distribute a training 
product with DaViTo. 

Leamability - 88 percent of subjects are either 
neutral or believe that more training is necessary 
to effectively be able to use DaViTo. 


Table 37. List of recommendations and the basis derived from analysis of the 
leamability portion of the survey results. 


6. Recommendations Based on Satisfaction Questions 

The satisfaction segment of the final survey generated only one unique 
recommendation, found in Table 38. Subjects desired that a help capability be designed 
into the DaViTo interface. Currently there is little help functionality, only a link to the 
OpenMap developer’s page, and no user’s manual is accessible from the interface. 
Recommendation C.16, to link the user’s manual to the help tab is a repeat of C.3. If the 
user’s manual provides proper guidance, this will increase user satisfaction. Similarly, 
recommendation C.17 recommends going further with a comprehensive help capability. 
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This feature would put DaViTo on par with more fa mi liar software packages that utilize 
searchable help databases in addition to online help capabilities. 

Additional recommendations in Table 38 are C.18 and C.19, call for improved 
software stability, and more graphing tools, respectively. These are repeats of A.l and 
C.4, respectively. Both recommendations were derived from participant free-form 
responses to satisfaction questions. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.16 

Link the user’s manual 
to the Help tab on the 
ribbon bar. 

Satisfaction - 46 percent of users think they can 
conduct basic tasks, and 25 percent of users think 
they can complete complex tasks, without the 
user’s manual. 

C.17 

Develop a 
comprehensive help 
capability for DaViTo. 

Satisfaction - Only 21 percent of subjects are 
comfortable using DaViTo without assistance. 
Additionally this was the second most frequent 
complaint with regards to satisfaction. 

C.18 

Improve software 
processing, response 
time, and stability. 

Satisfaction - This was the top complaint with 
regards to satisfaction. 

C.19 

Include graph tools to 
improve graph 
readability. 

Satisfaction - This was the third most frequent 
complaint with regards to satisfaction. 


Table 38. List of recommendations and the basis derived from analysis of the 
satisfaction portion of the survey results. 


7. Recommendations Based on Visualization Questions 

The visualization segment of the final survey generated only one unique 
recommendation, found in Table 39. There was some confusion among the users with 
regard to the identification of the icons discussed in recommendation C.23. Since both 
have similar functionality, it is recommended to use a common icon. 
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Additional recommendations in Table 39 are C.20, C.21, and C.22. These are 
repeats of C.4, B.2, and C.13, respectively. These recommendations were derived from 
participant free-form responses to visualization questions and are tailored to improve the 
interpretability of the displayed output data within DaViTo. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.20 

Include graph tools to 
improve graph 
readability. 

Visualization - Only 12.5 percent of subjects 
agreed that output data is clear to interpret, and 
the second most frequent response to desired 
changes to the output was more graph 
manipulation tools. 

C.21 

Add more options for 
different graph types. 

Visualization - The most frequent response to 
desired changes to software output was more 
graph options. 

C.22 

Make data types 
within a data set 
sortable by color. 

Visualization - Only 12.5 percent of subjects 
agreed that displayed output data is clear to 
interpret, and the third most frequent response to 
desired changes to the output was a greater 
capability to tailor the data. 

C.23 

Alter the cursor 
navigation button 

Visualization - 54 percent of subjects could not 
correctly identify the cursor navigation icon. 71 
percent of subjects could correctly identify the 
map navigation icon. Some incorrect answers for 
each were the other icons title. The functionality 
associated with these icons is very similar. 

R An 

j map 

icon. 

to match th( 
navigation 


Table 39. List of recommendations and the basis derived from analysis of the 
visualization portion of the survey results. 
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8 . 


Recommendations Based on Improvement Questions 


The final segment of the session three survey asked a series of questions as to 
how the subjects would improve DaViTo. Many of the recommendations are repeats of 
previous comments and were mentioned earlier with regards to specific usability 
attributes, so only new recommendations are discussed in Table 40. 

Recommendation C.24 was mentioned by multiple users, as a desire for standard 
two button mouse functionality. Software designed for Microsoft Windows based 
systems is typically developed in anticipation of a two or three button scrolling mouse 
being the interaction device of choice. In most cases, left clicking selects an object or 
carries out a function; right clicking brings up a menu of options. In DaViTo a specific 
converse to this was right clicking outside of a polygon in order for it to be drawn and 
populated with statistics. This additional click was problematic and confusing for some 
participants. 

Today applications such as Google Earth, MapQuest, and other mapping tools 
have a rich user interface. Not only are they more visually appealing, it also has 
functionality when trying to understand the terrain of the area you are looking at, and 
keeps users more engaged. Recommendations C.25, C.26, and C.27 in Table 40 will 
incorporate a 3D mapping capability, add contrast to the color scheme, and change the 
default data point color from gray to a color with increased contrast. This will improve 
the visualization of DaViTo, as well as its usefulness and satisfaction. 
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Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.24 

Standardize two 

button mouse 
functionality. 

Multiple Attributes - Follow the standard 
functionality for a two-button mouse that most 
software developers utilize. 

C.25 

Incorporate an area 
relief option or 3D 
capability similar to 
Google Earth. 

Visualization - Top response when subjects were 
asked what they would change about the color 
scheme or user interface layout. 

C.26 

The color scheme is 

bland. Add more 

contrast. 

Visualization - In an effort to look more inviting 
to users software should have visual appeal, even 
in military applications. 

C.27 

Change the default 
data point color. 

Visualization - The current default of gray has no 
contrast with the interface due to frequent use of 
different shades of gray. 


Table 40. List of recommendations and the basis derived from analysis of the 
improvement portion of the survey results. 


D. ADDITIONAL AUTHOR RECOMMENDATIONS 

Throughout the development of the experiment and observing subject sessions the 
authors of this thesis received numerous hours of exposure to DaViTo. Table 41 contains 
additional recommendations by the authors based on their observations and interaction. 

Recommendation D.l would increase the usable area of the software for data 
representation and analysis by allowing users to collapse the menu areas on the left of the 
user interface. Once the files are opened or the options in the segments of the interface 
on the left side of the screen are complete, they are no longer very useful and only 
remove work space from the user. D.2 and D.3 are intended to address future 
possibilities for the use of DaViTo as discussed with TRAC Monterey. It is desirable for 
the use of this software to spread beyond the realm of the small group of analysts it was 
designed for, and the inclusion of nautical capabilities would advance this. Training for 
the new target audience as well as advanced training for the initial audience would 
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increase usefulness for both parties. To get the tool in use beyond the army, 
incorporation of mapping features that benefit the other service branches would be 
helpful. 

Finally, recommendation D.4 addresses the development of a database for use 
with DaViTo as well as standardized data formats. The output of DaViTo is only as good 
as the information entered into it. It was stated in Data Visualization Tool that there are 
vast amounts of data that are not being properly analyzed and the subsequent results 
getting to decision makers (Evangelista, Henry, Buttrey, & Whitaker, 2012). Through 
conversations with TRAC Monterey there does not currently exist such a database. This 
should be developed concurrently with DaViTo to maximize its potential and ensure the 
tool is able to be utilized to its fullest potential. 
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Designator 

Recommendation 

Description 

Basis for Recommendation 

D.l 

Add a collapse 
capability to the Data 
Sets, Predefined 
Polygons, and Regions 
Selection List areas of 

the interface 

Visualization - This will expose more of the Map 
and Data Display area, reducing clutter and 
allowing the user more capability to work within 
the primary workspace of the interface. 

D.2 

Develop training 
products for both a 
basic user and 
experienced analyst. 

Usefulness - Recommendations C.l and C.2 
address the development of training products. 
There are complex statistical capabilities in 
DaViTo. An analyst with advanced skills would 
benefit from tailored training that goes beyond the 
capabilities in the user interface. 

D.3 

Extend mapping 
capability to include 
nautical charts. 

Usefulness - Many Navy subjects identified 
additional uses such as counter-piracy or sea- 
based counter-drug operations. 

D.4 

Develop a 

standardized Army (or 
service wide) database 
and data formats for 

use with DaViTo. 

Usability - Without a formal database and data 
formats the tool itself may suffer reduced 
usability. 


Table 41. List of author recommendations and the basis derived from the observations 
of subjects and interaction with DaViTo. 
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VI. CONCLUSION AND FUTURE WORK 


A. SUMMARY 

The Data Visualization Tool developed by TRAC Monterey is a powerful piece of 
software that has the potential to be instrumental in future military operations. If 
properly developed through continuous revision and improvement, it will see broader 
employment. The tool itself has wide-ranging application throughout the Army and 
potentially the entire military. 

This thesis was limited in scope to the usability of DaViTo, although there are 
other opportunities for research. A formal usability study was not conducted during the 
development of DaViTo and as such this thesis was done in lieu of that following the 
initial release. The primary objective was to provide actionable recommendations to 
TRAC Monterey that could be used to improve the usability of the tool through future 
revisions. 

Usability has many definitions, but in the context of this thesis it can be best 
described as “the quality of a system with respect to ease of learning, ease of use, and 
user satisfaction” (Rosson & Carroll, 2002). As a tool for the analysis of data with both 
spatial and temporal components DaViTo has the potential to be instrumental. Increasing 
the usability of DaViTo by improving upon all of its attributes will enhance that 
potential. 

The experiment itself was developed to rapidly train and familiarize participants 
with DaViTo. Once complete, gathering data from subjects was the next step. With 24 
volunteer participants over 70 man-hours were spent training, testing, and surveying 
them. Data was collected through eye-tracking while conducting tasks, inspecting 
graphs, and answering numerous survey questions. 

In order to compartmentalize collected data for analysis it was split into three 
groups. The first was the collection of data through eye-tracking. This data was useful 
for determining where subjects are focusing their attention while trying to complete a 
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task. The software used, EyeWorks Record, also collected mouse clicks and tracked the 
time elapsed from task to task. The second method of data collection was by presenting 
participants with multiple graphs, asking quesitons about them, and then analyzing their 
responses. The outcome of this portion of the thesis was useful at determining the best 
ways to present information to decision makers in a manner in which they can 
understand. Finally, a huge amount of data directly from the users was collected in brief 
questioning in the first two sessions, followed by a complex and thorough final surery in 
session three. This data was particularly useful for statistical analysis of questions related 
to the usability attributes identified in Table 1, as well as the development of 
recommendations directly from subject free-form responses. 

Methods to improve DaViTo is the heart of this research. Four subsets of 
recommendations were generated from the results of the data analysis. The first subset 
was derived from the results of subject tasks. Recommendations were developed that 
would improve usability in nearly all attributes. The second subset of recommendations 
were based on the results of subject analyses of five graph types. Data visualization and 
analysis within DaViTo was a large part of this thesis, and many improvements were 
suggested that would increase the usability and assist developers in meeting the initial 
goals outlined for the software. The next subset focused on three common questions 
across each session as well as an extensive final survey. With the wealth of information 
gathered, multiple new and unique recommendations were formulated to improve upon 
all attributes of usability. The final subset of recommendations were developed by the 
thesis authors and are a result of the hours spent designing and carrying out the 
experiment sessions and the accompanying research. They consist of four final big 
picture recommendations to assist developers with the next revision of DaViTo. 

In the end, 37 recommendations have been formulated for dissemination to TRAC 
Monterey. Several of these are repeats; however, they have a different usability attribute 
and basis for recommendation. The full list of recommendations can be found in 
Appendix A. Ideally, the fruits of this thesis will make it into future revisions of DaViTo 
and succeed in making it a premier software package useful throughout the military and 
beyond. 
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B. FUTURE WORK 


Prospective topics for additional research derived from this thesis are abundant. 
Usability testing and eye-tracking both have far-reaching capabilities in the future. The 
first is a follow-up usability test of the next revision of the DaViTo software. 
Verification of improvements in all attributes of usability would be useful to verify the 
recommendations resulting from this thesis were indeed applied and beneficial. Further 
research is also necessary in the field of data visualization. This thesis was very limited 
in scope with regards to the techniques tested. As data visualization methods advance 
there is a need for research in this field to continue evolving DaViTo. Additionally, the 
software could be further improved and refined through additional testing. Another is 
research into the application of eye-tracking to usability testing. As a relatively new 
source of data for research, eye-tracking has boundless potential. Determining the most 
effective ways to utilize the process and resulting data to maximize improvements to the 
research subject would be beneficial. Finally, usability testing of other products with the 
same methods used in this thesis would be useful. Confirmation of the methods used to 
conduct testing on DaViTo could lead to advancement in the field of usability testing and 
enhance future product analyses. 
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APPENDIX A. CONSOLIDATED RECOMMENDATIONS 


Appendix A presents a consolidated list of recommendations for the DaViTo 
software in Table 42. The designator represents which portion of results it came from. 
Recommendations based on task results have the designator A, those from graph results 
are designated B, the C designation is based on survey results, and finally the D 
designator is for author recommendations. The attributes are described previously in 
Table 1. Repeated recommendations are identified with an asterisk in the designator, but 
have a different attribute and/or basis. 


Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

A.l 

Improve software 
processing, response 
time, and stability. 

Efficiency - When drawing user-defined 
polygons there was excessive lag time or the 
software crashed. 

A.2 

Implement a progress 
status bar. 

Effectiveness - When drawing polygons, the lag 
time led subjects to continue clicking outside the 
polygon because there was no indication of 
progress. 

A.3 

Draw and populate a 
polygon when drawn 
by the user. 

Effectiveness - Right clicking outside of the 
polygon was counter-intuitive to users. 

A.4 

Implement file 
operations in the main 
menu ribbon. 

Effectiveness - Heat mapping shows that many 
users looked to the menu ribbon for file 
operations. 

B.l 

Make the Stacked Bar 
Format (SBF) the 
default graph type. 

Efficiency - The SBE graph was the most 
conducive to generating a correct answer from 
experiment subjects. 

B.2 

Add more options for 
graph types. 

Usefulness - Different graphs have better 
performance relative to the question asked. 
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Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.l 

Develop a tutorial 
based training product 
for DaViTo. 

Leamability - Currently there are no training 
products for DaViTo beyond a basic PowerPoint 
slide show. 

C.2 

Distribute a training 
product with DaViTo. 

Leamability - There is only a PowerPoint slide 
show distributed with DaViTo currently. 

C.3 

Link the user’s manual 
to the Help tab on the 
ribbon bar. 

Usefulness - Currently the user’s manual is not 
accessible via the DaViTo interface. 

C.4 

Include graph tools to 
improve graph 
readability. 

Usefulness - Graph interpretation was identified 
by subjects as the one of the most difficult tasks. 

C.5 

Make graphs easier to 

create. 

Usefulness - Graph creation was identified by 
subjects as the one of the most difficult tasks. 

C.6* 

Draw and populate a 
polygon when drawn 
by the user. 

Usefulness - Polygon functions were identified 
by subjects as one of the most difficult tasks. 

C.l 

Improve the software 
reaction time. 

Effectiveness - 83 percent of subjects were not 
satisfied with the software reaction time. 

C.8* 

Improve software 
processing, response 
time, and stability. 

Effectiveness - Subjects stated this as the worst 
deficiency that reduced the simplicity of use of 
DaViTo. 

C.9* 

Implement a progress 
status bar. 

Effectiveness - Subjects stated this as the second 
worst deficiency that reduced the simplicity of 
use of DaViTo. 

C.IO 

Include geographic 
labels in the Map and 
Data Display Area. 

Effectiveness - Subjects stated this as the third 
worst deficiency that reduced the simplicity of 
use of DaViTo. 

C.ll* 

Improve software 
processing, response 
time, and stability. 

Efficiency - 67 percent of subjects disagreed that 
the software was stable and not prone to crashing. 
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Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.12* 

Include geographic 
labels in the Map and 
Data Display Area. 

Efficiency - Subjects reported this as the top 
deficiency that made visual representations easy 
to read. 

C.13 

Make data types 
within a data set 
sortable by color. 

Efficiency - Subjects reported this as a deficiency 
that made visual representations difficult to read. 

C.14* 

Develop a tutorial 
based training product 
for DaViTo. 

Eearnability - 75 percent of subjects reported that 
most people could figure out how to use DaViTo 
with the tools provided; however 58 percent 
believe more training products are needed. 

C.15* 

Distribute a training 
product with DaViTo. 

Eearnability - 88 percent of subjects are either 
neutral or believe that more training is necessary 
to effectively be able to use DaViTo. 

C.16* 

Link the user’s manual 
to the Help tab on the 
ribbon bar. 

Satisfaction - 46 percent of users think they can 
conduct basic tasks, and 25 percent of users think 
they can complete complex tasks, without the 
user’s manual. 

C.17 

Develop a 
comprehensive help 
capability for DaViTo. 

Satisfaction - Only 21 percent of subjects are 
comfortable using DaViTo without assistance. 
Additionally this was the second most frequent 
complaint with regards to satisfaction. 

C.18* 

Improve software 
processing, response 
time, and stability. 

Satisfaction - This was the top complaint with 
regards to satisfaction. 

C.19* 

Include graph tools to 
improve graph 
readability. 

Satisfaction - This was the third most frequent 
complaint with regards to satisfaction. 

C.20* 

Include graph tools to 
improve graph 
readability. 

Visualization - Only 12.5 percent of subjects 
agreed that output data is clear to interpret, and 
the second most frequent response to desired 
changes to the output was more graph 
manipulation tools. 
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Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

C.21* 

Add more options for 
graph types. 

Visualization - The most frequent response to 
desired changes to software output was more 
graph options. 

C.22* 

Make data types 
within a data set 
sortable by color. 

Visualization - Only 12.5 percent of subjects 
agreed that displayed output data is clear to 
interpret, and the third most frequent response to 
desired changes to the output was a greater 
capability to tailor the data. 

C.23 

Alter the cursor 
navigation button 

Visualization - 54 percent of subjects could not 
correctly identify the cursor navigation icon. 71 
percent of subjects could correctly identify the 
map navigation icon. Some incorrect answers for 
each were the other icons title. The functionality 
associated with these icons is very similar. 

R An 

to match the map 
navigation icon. 

C.24 

Standardize two 

button mouse 
functionality. 

Multiple Attributes - Follow the standard 
functionality for a two-button mouse that most 
software developers utilize. 

C.25 

Incorporate an area 
relief option or 3D 
capability similar to 
Google Earth. 

Visualization - Top response when subjects were 
asked what they would change about the color 
scheme or user interface layout. 

C.26 

The color scheme is 

bland. Add more 

contrast. 

Visualization - In an effort to look more inviting 
to users software should have visual appeal, even 
in military applications. 

C.27 

Change the default 
data point color. 

Visualization - The current default of gray has no 
contrast with the interface due to frequent use of 
different shades of gray. 
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Designator 

Recommendation 

Description 

Attribute and Basis for Recommendation 

D.l 

Add a collapse 
capability to the Data 
Sets, Predefined 
Polygons, and Regions 
Selection List areas of 

the interface 

Visualization - This will expose more of the Map 
and Data Display area, reducing clutter and 
allowing the user more capability to work within 
the primary workspace of the interface. 

D.2 

Develop training 
products for both a 
basic user and 
experienced analyst. 

Usefulness - Recommendations C.l and C.2 
address the development of training products. 
There are complex statistical capabilities in 
DaViTo. An analyst with advanced skills would 
benefit from tailored training that goes beyond the 
capabilities in the user interface. 

D.3 

Extend mapping 
capability to include 
nautical charts. 

Usefulness - Many Navy subjects identified 
additional uses such as counter-piracy or sea- 
based counter-drug operations. 

D.4 

Develop a 

standardized Army (or 
service wide) database 
and data formats for 

use with DaViTo. 

Usability - Without a formal database and data 
formats the tool itself may suffer reduced 
usability. 


Table 42. Consolidated list of recommendations for the improvement of DaViTo and 
their basis derived from all research methods. 


95 





THIS PAGE INTENTIONALLY LEET BLANK 


96 



APPENDIX B. ADDITIONAL TASK RESULTS 


Appendix B contains cumulative gaze frequency data for all final session tasks as 
heat maps in Figures 19 through 26. Figures 27 through 36 show the cumulative counts 
of gaze observations for all subjects in each region of the interface for the heat maps. 
Heat maps and gaze observation graphs presented in the body of the thesis are not 
reproduced in this Appendix. 
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Figure 19. Final session task two heat map and mouse clicks. Task two directed subjects to load the Kenya Roads Shapefile, which 
required interaction with the Shapefiles region. 
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Figure 20. Final session task three heat map and mouse clicks. Task three directed subjects to go to the Africom view. 
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Figure 21. Final session task five heat map and mouse elicks. Task five direeted subjects to close and delete the chart generated in 
task four. 
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Figure 22. Final session task six heat map and mouse clicks. Task six directed subjects to hide the Map Controls ribbon bar. 
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Figure 23. Final session task seven heat map and mouse elicks. Task seven direeted subjeets to navigate to Iraq without the use of 
the Map Controls portion of the interfaee. 
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Figure 24. Final session task eight heat map and mouse clicks. Task eight directed subjects to go back to the previous view. 
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Figure 25. Final session task nine heat map and mouse clicks. Task nine directed subjects to remove all data sets and Shapefiles, then 
load the Indure_Demo data set and the afghanistan_provincial_boundaries Shapefile. 
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Figure 26. 


Final session task eleven heat map and mouse clicks. Task eleven asked subjects to answer a trend analysis question using 
the graph generated in the user defined polygon from task ten. 
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Gu* Observations In Each Region 



Figure 27. Graph of final session, task two, cumulative counts of gaze observations for 
all subjects. Task two directed subjects to load the Kenya Roads Shapefile, 
which required interaction with the Shapefiles region. Counts correlate to the 
number of subject fixations in each region shown in Figure 19. 


Gaze Observations in Each Region 



Figure 28. Graph of final session, task three, cumulative counts of gaze observations for 
all subjects. Task three directed subjects to go to the Africom view. Counts 
correlate to the number of subject fixations in each region shown in Figure 
20 . 
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C«z« Observations in Each Region 



Figure 29. Graph of final session, task four, cumulative counts of gaze observations for 
all subjects. Task four directed subjects to create a user defined polygon and 
display the chart. Counts correlate to the number of subject fixations in each 
region shown in Figure 9. 


Gaze Observations In Each Region 



Figure 30. Graph of final session, task five, cumulative counts of gaze observations for 
all subjects. Task five directed subjects to close and delete the chart 
generated in task four. Counts correlate to the number of subject fixations in 
each region shown in Figure 21. 
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Cue Obs*<vation$ In E*ch R«fh>n 


Figure 31. 


Figure 32. 



Graph of final session, task six, cumulative counts of gaze observations for 
all subjects. Task six directed subjects to hide the Map Controls ribbon bar. 
Counts correlate to the number of subject fixations in each region shown in 
Figure 22. 


Gue Obs*fvatiom In E*ch R*fton 



Graph of final session, task seven, cumulative counts of gaze observations for 
all subjects. Task seven directed subjects to navigate to Iraq without the use 
of the Map Controls portion of the interface. Counts correlate to the number 
of subject fixations in each region shown in Figure 23. 
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&B2* Observations In Each Raf ion 



Figure 33. Graph of final session, task eight, cumulative counts of gaze observations for 
all subjects. Task eight directed subjects to go back to the previous view. 
Counts correlate to the number of subject fixations in each region shown in 
Figure 24. 


Gaza Observations In Each Refton 



Figure 34. Graph of final session, task nine, cumulative counts of gaze observations for 
all subjects. Task nine directed subjects to remove all data sets and 
Shapefiles, then load the Indure_Demo data set and the 
afghanistan_provincial_boundaries Shapefile. Counts correlate to the 
number of subject fixations in each region shown in Figure 25. 
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G*2* Obs*rv«tion$ In E*ch Region 



Figure 35. Graph of final session, task ten, cumulative counts of gaze observations for 
all subjects. Task ten directed subjects to answer a question that required the 
creation of a user defined polygon, and to display the chart. Counts correlate 
to the number of subject fixations in each region shown in Figure 10. 


C«z* Observations In Each Rafion 



Figure 36. Graph of final session, task eleven, cumulative counts of gaze observations 
for all subjects. Task eleven asked subjects to answer a trend analysis 
question using the graph generated in the user defined polygon from task ten. 
Counts correlate to the number of subject fixations in each region shown in 
Figure 26. 
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APPENDIX C. ADDITIONAL GRAPH RESULTS 


Appendix C contains additional results from the analysis of the data 
representation portion of the experiment. Table 43 presents the data representation 
questions asked about each graph type. The statistical analyses of the questions in Table 
43 are presented in Tables 44 through 50 and Figures 37 through 43. 


Question 

Designation 

Session 

Question Text 

A 

One 

What were the total events for December 2012? 

B 

One 

What event had the highest frequency in Jun 2013? 

C 

One 

What month had the most events? 

D 

Two 

What month had the least events? 

E 

Two 

Which month had the highest frequency for event D? 

F 

Three 

Compare the months of December 2012 and December 

2013. What was the increase or decrease in total events? 

G 

Three 

What event had the highest sum for the months of February 
2012 and February 2013? 


Table 43. List of questions for each session and their designation letter. This is a repeat 
of Table 12 in Chapter IV, Section B, shown here to correlate data to 
questions. 


Question 

Accuracy 

Time 

A 

(% Correct) 

(sec) 

CCF 

41.67 

82.30 

SBF 

70.83 

34.90 

LCF 

33.33 

40.59 

SLMF 

8.33 

46.19 

SAF 

33.33 

35.56 


Table 44. Statistics for session one, question A, for all graph types. 
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Figure 37. Plot of session one, depicting time to answer versus accuracy of the answer 
for question A. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner. 


Question 

Accuracy 

Time 

B 

(% Correct) 

(sec) 

CCF 

75.00 

26.91 

SBF 

79.17 

25.77 

LCF 

58.33 

23.69 

SLMF 

12.50 

21.34 

SAF 

41.67 

16.29 


Table 45. Statistics for session one, question B, for all graph types. 


Session 1 , Question B Time vs Accuracy 

Accuracy (% correct) 
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Figure 38. Plot of session one, depicting time to answer versus accuracy of the answer 
for question B. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner. 
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Table 46. 


Figure 39. 


Table 47. 


Question 

Accuracy 

Time 

C 

(% correct) 

(sec) 

CCF 

45.83 

65.78 

SBF 

91.67 

22.77 

LCF 

50.00 

50.85 

SLMF 

45.83 

33.00 

SAF 

79.17 

22.70 


Statistics for session one, question B, for all graph types. 


Session 1 , Question C Time vs Accuracy 

Accuracy (% correct) 
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Plot of session one, depicting time to answer versus accuracy of the answer 
for question C. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner. 


Question 

Accuracy 

Time 

D 

(% Correct) 

(sec) 

SAF 

83.33 

28.27 

SLMF 

87.50 

24.30 

LCF 

25.00 

61.37 

SBF 

87.50 

20.85 

CCF 

79.17 

40.00 


Statistics for session two, question D, for all graph types. 
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Figure 40. 
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Plot of session two, depicting time to answer versus accuracy of the answer 
for question D. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner. 


Question 

Accuracy 

Time 

E 

(% Correct) 

(sec) 

SAF 

66.67 

40.97 

SLMF 

58.33 

34.32 

LCF 

75.00 

24.21 

SBF 

41.67 

36.92 

CCF 

87.50 

20.77 


Table 48. Statistics for session two, question E, for all graph types. 


Session 2, Question E Time vs Accuracy 

Accuracy [Yq correct) 
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Figure 41. Plot of session two, depicting time to answer versus accuracy of the answer 
for question E. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner. 
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Question 

Accuracy 

Time 

F 

(% Correct) 

(sec) 

LCF 

12.50 

107.13 

SLMF 

33.33 

67.39 

SAF 

50.00 

49.07 

SBF 

75.00 

44.33 

CCF 

50.00 

59.10 


Table 49. Statistics for session three, question F, for all graph types. 


Session 3, Question F Time vs Accuracy 
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Figure 42. Plot of session three, depicting time to answer versus accuracy of the answer 
for question F. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner. 


Question 

Accuracy 

Time 

G 

(% Correct) 

(sec) 

LCF 

87.50 

67.50 

SLMF 

62.50 

52.93 

SAF 

70.83 

42.31 

SBF 

75.00 

33.86 

CCF 

100.00 

50.15 


Table 50. Statistics for session three, question G, for all graph types. 
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Session 3, Question G Time vs Accuracy 
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Figure 43. Plot of session three, depicting time to answer versus accuracy of the answer 
for question G. Note the decreasing time on the Y-axis. Top performance is 
in upper right corner 
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APPENDIX D. ADDITIONAL SURVEY DATA 


A. THE THREE COMMON QUESTIONS 


Appendix D, part A, presents additional results from the three questions eommon 
to each session. The results, shown in Figures 44 through 48, are for each individual 
question. Note that the answers to the free-form questions in Tables 51 through 53 are 
not altered in content from what the subjects entered in the survey. 


Session 1 Question 1: How would you 
rate your first training session? 



Figure 44. Session one, question one, with responses on a scale of one (very poor) to 
five (very good.) 


Figure 45. 


Session 1 Question 2: How many 
more training hours are needed? 
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Session one, question two, subject’s self-evaluation of additional training 
hours needed. 
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Session One, Question Three: What were your top three likes and dislikes with 

your first exposure to DaViTo? 

Likes: Visual display of data possible for "walk through" with a boss, it seems like 
you could create and share databases with others to share the same view of a situation, 
shapes/displays/colors are customizable. 

Dislikes: It might be chugging through a lot of data when you move the map each 
time: seems to go more slowly than necessary. Otherwise, need more exposure to 

make a better judgment. _ 

Likes: 1) Easy to use 2) The tab identifies are descriptive 3) Quick to learn 
Dislikes: 1) Couldn't remember the zoom in 2) The distance circle is deceiving 3) 

Map changes slowly _ 

Likes: 1. Easy to navigate 2. Overlays 3. Quick reference 

Dislikes: 1. Seems like it runs (scrolls) slowly 2. Unable to directly click on data 

points when using cursor3. No city or population center overlays. _ 

Likes: All of the data is displayed in one window making it easy to use. Good 
introductory tutorial. Tool for zooming in on certain areas of interest is useful 
Dislikes: Lag time in updating the map. Map controls would be more useful if on the 

map. Zoom and pan functions could use more fine controls. _ 

Likes: Good Company- Eun presentation- Cool gadgets. 

Dislikes: Slow gui- no map labels- distance tool should have separate way points _ 

Likes: Simple navigation toolbar- saved views- 

Dislikes: Measurement tool graphics leave an artifact sometimes- slow refresh rate- 
Likes: 1. Easy to use 2. Easy to understand the interface 3. Redundant ways to 
accomplish the same task 

Dislikes: 1. No preloaded options for US layers (geographical boundaries) 2. Not clear 

on how to add/create additional layers 3. No built-in tutorials _ 

Likes: 1 - common menus and icons to other similar software - easier to learn 2 - 
simple- plain interface. 3 - easy to get to the right functions. 

Dislikes: 1 - speed of map refresh - likely a function of the hardware- but it was a little 
annoying to see the lag between a selection that the new map 2 - "MGRS" was 

misspelled in the dropdown menu - was written "MRGS" _ 

Likes: 1. My first like is the usability of the navigation functions. They are very easy 
to understand and apply. 2. My second like is the use of a importation of data sets 
from pre-existing files. 3. My third like is the ease of understandability of how to use 
the program. 

Dislikes: 1. My first dislike is the time lag in the zoom in- zoom out function. 
2. My second dislike is the caliper function of the use of distance. This tool needs to be 
updated to something that is more user friendly and easily shows data that 

you are trying to find to new users. _ 
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Session One, Question Three: What were your top three likes and dislikes with 

your first exposure to DaViTo? 

Likes: 1. Liked that it was pretty simple to use. 2. There wasn't too much information 
(overload) on the screen. 3. It also seems familiar with other applications (like GPS) so 
that made it easy to figure out. 

Dislikes: l.I didn't like that it seemed slow to respond. I feel like I could easily over 
task the computer if I was trying to get information quickly. 2. It took me a second 
(with help) to figure out how to use the measurement tool. 3. While I liked that there 

wasn't information overload- at the same time I felt it was too generic and simple. _ 

Likes: I liked how the tools were easy to see and use on the left panel. I've have 
experience with other military mapping programs and they have always confused me 
on how mapping data is loaded and accessible by the user. 

Dislikes: I don't like how slow the refresh rate it. I would get frustrated using this 
program in a stressful environment when I would start clicking to many times and get 

into a "do-loop" because the program was too slow. That is about it. _ 

Likes: Multiple options to manipulate - redundancy- user interface is intuitive- color 
are not distracting and appropriate 

Dislikes: System latency- zoom option (the squared icon on the tool bar) when 
selecting a location hard to center- no country labels provided (similarly- states- 

provinces- districts- etc.) _ 

Likes: Easy to see the maps on the right display. Once I learned the navigation 
portion- it was easy to zoom in on a specific area. Easy to load data sets. 

Dislikes: I got confused on zooming in with the mouse scroll; I went the opposite 

direction. _ 

Likes: 1) Eairly intuitive 2) Efficient selection of functionality (no more a couple 
clicks to do what I wanted) 3) Simplicity (functionality I was instructed in during this 
session was all visible on the screen) 

Dislikes: 1) Didn't like the position of lat/long and distance down in the left corner 
(my preference would be maybe in the top middle- closer to where I was looking 
(ideally the distance would show up in the middle of the great circle distance as you 
measure) 2) Zoom box function was refined (it worked but didn't allow me to drag 
down to the lower right like I intended) 3) Appears to have limited level functionality 

compared to something like CPOE (where you could analyze in 3D) _ 

Likes: Easy to understand icons (intuitive graphical representation -i- mouse over 
tooltip) 2: Clean interface- no clutter 3: Controls are similar to normal software (e.g. 
pan using click and drag- zoom using selection box or mouse wheel) 

Dislikes: 1: Response Eag- especially when click-and-drag to pan when data sets are 
loaded 2: Text is a bit small (necessary tradeoff to maximize map display area?) 3. 
Info bar (where it shows the Eat/Eong/Distance) at bottom left is small and tucked 
away. Hard to use with the distance measurement function. _ 
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Session One, Question Three: What were your top three likes and dislikes with 

your first exposure to DaViTo? 

Dislike: Information could be displayed in conjunction with a map built in. Did not 
like the system being "jittery" when navigating on the map. Could lose the location 
you are looking for when moving around do to the slow response from mouse input. 
Needs to be smoother. Slow, No country labels, Distanee ring to thin. Distance tools 
doesn't allow you to make separate waypoints 

Likes: Easy to use. _ 

Likes: 1) Multiple map projections available. 2) Clearly defined data set types along 

with data set previews. 3) Ease of data set integration into map. _ 

Likes: 1. Ease of use. Resembles other map programs. 2. Ability to quickly toggle 
between chosen projections without using the zoom feature. 3. Theater drop down to 
focus on the different AORs. 

Dislikes: 1. Distance tool line blends in with rest of map lines. Not distinguished. 
Likes: Ease of use. Convenient lay out. 

Dislikes: Eack of labels _ 

Likes - 1.Simple menus- 2. Quick feedback when a tool was used (distance- zoom) 

Dislikes - 1. not sure yet _ 

Likes: user friendly dislikes: when using the pan tool the delay caused by moving the 

map _ 

Likes: 1) Very user friendly 2) Good graphical user interface 3) Eollows standard 
format like most Windows programs- so if you are a Windows user it was easy to 

understand. Dislikes: 1) A little slow with zooming out _ 

Likes: Simplicity- Eamiliarity- distance measurement 

Dislike: Scaling back out- no scroll wheel use- having to come up with three dislikes 
Likes: Ease of use. Ease of understanding of the functionality of the tool. Simplicity 
of functions 

Dislikes: Eatency of program reaction to user demand D: Eayout of Screen D: Default 
color scheme; should have used Army colors _ 

Table 51. Table of likes and dislikes for all 24 subjects in session one. Comments 
taken directly from the EyeWorks software with no modification. 
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Session 2 Question 1: How would you 
rate your second training session? 


.Si. 

S' 

3 


u 

.a 

E 

3 

z 


14 

12 

10 

8 

6 

4 

2 

0 


I Response 



Figure 46. Session two, question one, with responses on a scale of one (very poor) to 
five (very good.) 


Figure 47. 


Session 2 Question 2: How many 
more training hours are needed? 
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Session two, question two, subject’s self-evaluation of additional training 
hours needed. 
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Session Two, Question Three: What were your likes and dislikes with your 

second exposure to DaViTo? 

Likes: Overall like the product and potential. 

Dislikes: The functionality associated with the shapes gave me a bit of trouble 
(following the tutorial and associated "outside the circle left-click" didn't work at first. 

Likes: Easy to understand. Helps to have the tutorial run through the steps. Would 
take more time using it to get all the steps correct. Just more practice time on the tasks 

that need to be accomplished. _ 

Dislikes: I did not like that the graph display did not have an easy method of changing 
the axis increments. The recall of graphs also seemed to take a little effort. 

Likes: The display of the charts for the different regions is helpful. 

Dislikes: The right click function did not work to display the full menu on several 
occasions- on the map and the chart view. It would be useful to have an hourglass to 
show progress when loading data sets. 

Likes: I enjoyed learning more of the capabilities of DaViTo 

Dislikes The right-left mouse use when creating and exploring charts 

Likes: Liked the way you could get to a graph so quickly based on just drawing a 

Shapefile on the chart. Very convenient. 

Disliked: The way you had to repeatedly right click on the smaller graphs to bring up 
the options menu. Initially- all that is available is the create option (first four to five 

clicks). _ 

Likes: Easy to use 2. Shape tools to select particular regions were intuitive. Chart 
output was easy to access and read 

Dislikes: 1. No option for irregular shapes 2. Data analysis tools not explained in- 

depth_ 

Likes: Seems like an easy way of embedding data into a map to make it more 
meaningful. 

Dislikes: The lag between mouse clicks and the menu appearing is frustrating- but not 
as frustrating as the Right click menus not acting the same each time. Sometimes I 
would get a short list of options- other times a longer list of options. This should be 
more consistent. 

Likes: I do like the capability of showing terrain on the graph and also the function of 
showing data per data "dot". 

Dislike: Also- loading time is still slow. I think that exposure for more than 15 
minutes is required to gain a full grasp. 

Dislikes: It was quite a bit harder for me to navigate through the charts- but with 
repetition I should be fine. So I disliked having to play "seek and find" to figure out 

what I needed. I really don't have any likes at this time. _ 

Likes: liked the way you can bring up data on specific events. 

Dislikes: I didn't like the slowness of the program. There appears to be decent amount 
of lag. Overall- it seems cluttered- lots of stuff to see and not very well organized. I 
don't see how anyone can get anything out of the thumb nails of the graphs that appear 
on the map. Plus they occlude the information under them. 
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Session Two, Question Three: What were your likes and dislikes with your 

second exposure to DaViTo? 

Like: Intuitive to use select a specific area and use shapes to draw graphs. The 
different options to modify plots (e.g. - freq vs percent- normalized- etc.) 

Dislike: Since there is an inherent lag in the system- a user feedback prompt or in¬ 
progress bar would be helpful to the user. Inability to change the line colors in the 

plots _ 

Likes: Reviewed previous Session- so that quickly got me reoriented to the tool. 
Gained knowledge on the system to include informational/statistical plots. Focused 

was on the different views and map manipulation. _ 

Likes: Clear instructions- good examples of tasks. Hands on is definitely a benefit to 
learning the application rather than just watching the video. 

Dislike: Is really my personal preference to see the instructions while I'm executing 
the task. Since I don't consider myself to be proficient with this software yet- I'd like 
the option to refer to the question you're asking as I go down the menus (to confirm I 
understand the task). 

Like: Step by step instructions 

Dislike: Need to memorize each step of task (the white screen) and no able to go back 

to reread the instruction. _ 

Dislikes: Still a little difficult to remember how to load data set. May have been to the 
brief exposure. 

Likes: Ease of general statistical analysis- access to overlays- ability to create new 
graphs on the fly. 

Dislikes: Response times to clicks- load times for data sets- unclear chart option 
explanations in control panel- too many steps to draw a shape. 

Like: The ability to create the chart in order to pull data. 

Dislike: Remembering the steps to create the chart. 

Dislikes: The maps take a while to upload- seems okay for this environment but it may 
be too restrictive in a real work environment that requires more changing- uploading 

Dislikes: Maps and charts loaded a little slow- but other than that no issues. _ 

Likes: Good graphical user interface- and the pop-up menus were easy to follow. 
Graph displays were good. No dislikes with my second exposure. 

Likes: Visual interface 
Dislike: Area selection method 

Dislikes: Deeper you go into the functionality the more you have to remember. Tough 
to do when you get older and your short term memory begins to slip. 

Table 52. Table of likes and dislikes for all 24 subjects in session two. Comments 
taken directly from the EyeWorks software with no modification. 
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Figure 48. 


Session 3 Question 2: How many 
more training hours are needed? 



Session three, question two, subject’s self-evaluation of additional training 
hours needed. 


Session Three, Question Three: What were your likes and dislikes with your third 

exposure to DaViTo? 

Likes: The interface works well for navigation and selection 

Dislikes But some of the analysis through the one chart I know how to make was 
difficult. If there is an icon that indicates the system is processing data- I don't know 
where it is and so would continue to click on different icons with no response._ 

Dislikes: Just don't remember enough of the tasks. It is laid out in a familiar fashion. But 
some of the later tasks-1 just couldn't remember how to do. 

Dislikes: Program responded sluggishly. An easier way to access chart information 
would be beneficial. Perhaps a chart display addition to the tool bar. Count functions 
would also be useful (i.e. count occurrences between this date and that date). 

Dislikes: The chart for lED attacks was difficult to read- it was hard to tell which time 
frame corresponded to the year and it was hard to trace a data point to a number on the y 
axis. There was no indication that the polygon was loaded and then there was a long 
delay if the polygon was selected before it was ready. 

Likes: Fun 

Dislikes: Like playing with the software-1 did not like the lack of progress display._ 

Dislikes: Some menu options are in counterintuitive locations (opening Shapefiles and 
map layers). I couldn't remember how to create one of the graphs- and searching the 
usual areas didn't give any clues. Once the graphs are readily available- the overlay onto 
a geographic area is convenient._ 

Likes: Simple instructions Tasks were well defined 

Dislikes: Country not labeled wrt task given (I didn't know where Kenya was) 2. No 
fantastically narrated training session by LT CutlerS. Couldn't figure out how to change 
the display of chart from individual event line to cumulative bar. 
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Session Three, Question Three: What were your likes and dislikes with your third 

exposure to DaViTo? 

Likes: Similar interface as other C2 systems in the Army (but difficult to learn the first 
time). 

Dislikes: The long load times with any indicator of what the computer was doing led to 
some mistakes and frustration. Overall- seemed slow. For the geographically- 
challenged- adding location names would be helpful. Such as labeling the nation names. 
Dislikes: Slow for loading 

Dislikes: It sometimes took a while to load maps. And after it loaded the maps I was 
unsure that it had completed. This may just be due to a slow processor. Additionally- the 
response to my right click was sometimes unresponsive or took too long. So it didn't feel 
right and I would have to try another way/method to accomplish a task. I also didn't like 
that when I was to load a shape of some sort- I needed to look for the polygon. That 
didn't make sense to me and it took me a bit (with some prompting) to get where I 
needed to go. As for likes- 

Likes: I seemed to be able to navigate better since this was my third session. But there 

still is a learning curve. _ 

Dislikes: SLOW!!I spend too much time "swimming". Did they write this thing in 
visual basic? Learn to code- code monkeys. 

Dislikes: User feedback indicating loading in-progress, a progress bar would improve 
user interface When loading a new data set or Shapefile the system does not pan to that 
location automatically When using the right click in the map option- the system displays 
different menu option some more complete that others but it is not clear to the user 
where to select in the map to get the right options 

Dislikes: 1. It takes too long to load the data sets. 2. If the system is working- there 
needs to be a message that states it is doing so. Still easy to fumble through/navigate 
-i-Charts = Awful - 

Likes: Once loaded with data- the tool provides very detailed charts for several types of 

data. Although small- the charts were easy to read based on color-coding. _ 

Dislikes: Time to process the data was a little confusing as to whether I had messed up 
or the processing was taking long/computer froze? Country labels would have been 
helpful with the boundaries (maybe put some hot spot labels when you hover over each 
province/country). When trying to hide the map controls I reverted back to a "Windows 
mentality" looking for -i- or - buttons to minimize- or hide- the map controls. Multiple 
clicks were probably the biggest concern I had during this session- since the mouse icon 
still moved. It lead me to doubt if I had selected the proper command(s). 

Likes: Interface is still manageable- despite forgetting some of the commands it is 
possible to try out and read the tool tips to recall what is the correct action to take. 
Dislikes: Slow to load data- perhaps good to have a progress bar so that user knows it’s 
still working on it (and estimate of how much longer) and not that the system is hanging, 
not able to zoom in on charts (e.g. to see just the 2010 data on lEDs)- making it hard to 

read off and count the values _ 

Dislikes: Slow as hell. Unresponsive. Very frustrating. Shouldn't take that long to 
gather data. Forgot how to navigate my way around the system. Couldn't remember how 
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Session Three, Question Three: What were your likes and dislikes with your third 

exposure to DaViTo? 

to draw a chart. 


Likes: I remembered most of the main features and did not have to be retrained 
Dislikes: Slow system response times 

Likes: Seemed easier to use by the third trial. Most tasks were relatively easy to recall. 
Dislikes: Difficult to right click open the menu for the data chart. 

Dislikes: Program too slow to respond 
Dislikes: Loading the map shapes was very slow 

Likes: It took a few minutes to remember how to navigate around but it seemed to come 
back to me. Good user interface. 

Dislike: Is the program is a little slow with generating charts. _ 

Like: None. 

Dislike: Not using the map control bar _ 

Likes: Ability to analyze data in picture format to gain insights. 

Dislikes: System latency in loading data. Inability to know whether system is working 
or not i.e. progress bars or hour glass that indicates that system is working on providing 
feedback for desired input. _ 

Table 53. Table of likes and dislikes for all 24 subjects in session three. Comments 
taken directly from the EyeWorks software with no modification. 


B. FINAL SURVEY 

Appendix D, part B, presents additional results from the final session survey free¬ 
form questions. Note that the responses in Tables 54 through 64 are not altered in 
content from what the subjects entered in the survey. 


_ Q: (Effect_3) Was the software simple to use? Please explain. _ 

Some of the finer controls associated with selection suffered due to lag. _ 

Yes. _ 

It was relatively easy to use but chart generation seemed more complex than it needed 

to be. _ 

Yes- with the exception of time delays without status icons to let the user know that 

information was being processed by the system. _ 

Yes- with training _ 

The basic interface is simple enough- but some options are harder to find than others- 

therefore they are more difficult to remember from one task to the next. _ 

Yes- mostly. Again- the chart output and geographical labeling were issues. Other 
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Q: (Effect_3) Was the software simple to use? Please explain. 

than that- it was fairly intuitive. 

Yes- but the lag created errors and mistakes 

The software was a little more user friendly with the video tutorial. If I had to use 
DaViTo without the tutorial-1 would have been lost! 

Generally it was okay. It seemed pretty simple at first- but I think I would get really 
frustrated if I had to use this a lot. There is definitely room for improvement. 

Yes. Things did what they looked like they would do. 

After three sessions it became easier to user to use. 

Yes- the software was easy to use. Again- the delay in loading the map data was the 
only issue. 

It was relatively intuitive and not complicated to operate 

While the execution of commands and navigation through menus was simple enough- 
the analysis features (i.e.; graphs) were sometime difficult to interpret. I think this 
could be solved by adjusting the formatting. 

Yes- controls are intuitive and tooltip pop-ups were helpful. 

No it sucked 

It was simple to use but it could definitely be improved with more integration of map 
navigation into the mouse controls. Separate buttons on screen for panning- zooming- 
grabbing- and data selection was a terrible GUI choice. 

Eor the most part. To be proficient would require daily use in order to not forget steps 
and functions. 

Somewhat 

Eike using my desktop at home 

Yes- simple user interface. 

Simple- yet not always intuitive 

Mostly intuitive. More use would have made it fully intuitive. 


Table 54. Table of free-form responses to question Effect_3 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


_ Q; (Eff_4) Was the visual representation easy to understand? _ 

Of geographical data and selections - yes. Of numerical data representations - no. 

Yes. Something to consider is my level of prior training. I was in the 4th Infantry 
Division in 1998 as we were testing EBCB2. So I have had a lot of exposure to both 
that and BET. I understand most of this set up from those two systems. The charts 

configuration was new to me. _ 

Yes _ 

Sorting of the data points within a data set with different colors could be helpful. 

Yes _ 

Yes- for the most part. Some option locations were inconsistent with other similar 

mapping GUIs. That makes searching around for it difficult. _ 

Yes- for the most part. The charts and lack of geographical labeling were really the 
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Q: (Eff_4) Was the visual representation easy to understand? 

major problem. 

Yes 

Yes 

If we're talking maps- then yes. If we're talking graphs- NO. The general display 
wasn't hard- but I wouldn't say it was really easy to understand. It took some training 
to get used to it. 

Yes 

Yes however- labels should be provide 

Yes- no issues with the actual visual portion- minus the charts. 

Relatively so 

Of the maps and controls yes- but not the charts (as I stated previously- the line charts 
were a pain to read) 

Yes 

Yes 

Projecting the charts directly onto the regions increased clutter on the screen and is a 
bad stylistic choice. The map projection was clunky as well- and the software would 
be wise to use a Google Earth type 3d view vice a 2d projection. 

Yes. Depending on the level of detail- labels of locations would be useful. 

map- yes, graphs- no 

Yes! 

Yes except for some of the charts (the cluster charts were the worst I think). 

Yes- except the charts 

Yes 


Table 55. Table of free-form responses to question Eff_4 for all 24 subjects in session 
three. Comments taken directly from the EyeWorks software with no 
modification. 


_ Q:(Use_2) What was the most difficult task to complete? _ 

Selection of individually highlighted areas and the graphs associated was sometimes 
difficult due to the need to "click inside - wait - click outside" delays. I found myself 

wondering if I had done it correctly based on the lag in response. _ 

Setting up the charts. I definitely needed more time using that. I know that was a focus 

last session. But I was not practiced enough to remember it correctly. _ 

Graph generation _ 

Reading lED attacks in 2010 from the chart. _ 

Navigate with map tools hidden. _ 

binding the chart function for the Kenyan data. _ 

Reading the chart output. _ 

Go to a previous view. _ 

Creating the charts _ 

Deciphering the graphs to get usable information. Way too difficult. I'm still not sure 
if I read the graphs right. _ 
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Q:(Use_2) What was the most difficult task to complete? 

The hardest part was remembering what the actual question was. Everything else 
seemed pretty straight forward. 

None 

Hiding the map controls. 

Reading the ridiculous graphs 

Eine chart interpretation was the most difficult since everything is so closely spaced 
and you have to look over to the left to judge # of events. 

Reading off number of lED attacks (because I forgot some of the steps- and because 
the chart was small) 

Reading the graphs. 

Setting up a graph for specific regions- since it took a while and the user is not sure if 
the system is responding or still waiting for an input. 

Making and opening the chart. 

binding the number of lED attacks in Afghanistan 

creating the rectangle shape 

Drawing the polygon around the northwestern provincial district to generate the chart. 

Chart selection 

Reading the graphs 


Table 56. Table of free-form responses to question Use_2 for all 24 subjeets in session 
three. Comments taken direetly from the EyeWorks software with no 
modification. 


Q: (Satisfaction_7) Were you satisfied with the performance of the software? 
_ Please explain. _ 

I would like to know when it is processing data (to know to wait). If analysis tools 
(charts and data displays) were offered with visual examples of what the products 
looked like- it might assist choosing the correct one for the job. Otherwise- the 
software seems like it would be a reasonably good tool for both analysis and visual 

display of data. _ 

Not with my own performance- no. If I had to use this system on the move-1 would 
not be happy with my current level of training. I would need much more time using 
the system. If this was for tactical operations center-1 would be able to use it better. 

Be able to also input data easier. _ 

Based upon the questions posed and my background as a Surface Warfare Officer 
this software would be highly useful if data could be displayed in maritime regions. 
With this software- or similar software and good historical data and piracy and 
counter narcotics missions that I have performed could have been effective if we had 

the data and tools (software) to effectively focus our limited resources. _ 

Overall it has a lot of lag and lacks fine controls. The graphs are hard to read- but 
there may be functionality to make them more readable. With some improvement 

this could be a useful tool for analysis of geographic data. _ 

Yes- but more training is needed and adjustments to the software would be 
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Q: (Satisfaction_7) Were you satisfied with the performance of the software? 

Please explain. 

beneficial. 

Yes- with more frequent use and a basic help menu- this would be useful software. 

Somewhat. I feel like this software could be used effectively with training or better 
software design. I recommend including up-to-date geographical information with 
the software. The information should be able to be updated automatically and 
manually. This way- it could be updated automatically with a steady internet 
connection or manually if on deployment or in the field. Also- the graph display 
could be more intuitive. There should be an easier way to change the display of the 
data (line- bar- pie- etc.) by right-clicking on the displayed chart. The analytical 
tools that are included should be addressed in training or in the user's manual. They 
should be written so a person with a sixth grade reading (and thinking) level could 
understand them. 

Somewhat. If the 'dislikes' were fixed- it would be very helpful. 

I feel the software is a great tool- however- without a larger knowledge base of how 
to run the software-1 would be at a disadvantage. 

No- I had the worst time deciphering the graphs. I feel that I could manipulate 
DaViTo well enough to get the graphs- but it's ridiculous to get any usable data from 
the graphs. 

No- too slow. And in general the graphs aren't very good. Seems to be too much 
information at once. 

Moderately satisfied- while the tool is simpler to use that ArcGIS it is very limited 
requiring improvements such as user interface improvements to provide feedback to 
the user- including labels for maps- improving system latency- etc. 

I did not like the amount of time that it took to load the data sets. If there was a 
notification that stating loading- then maybe it would have been ok. 

Same comments as previous. Relatively easy to use software. I could pick it up 
where I left off each session with only a few seconds of menu searching at the most 
to re-familiarize myself with the tools. The graphs- however- are mind-numbingly 
bad. They are difficult to read. If they were as intuitive as the basic controls for the 
rest of the software- I would recommend this program wholeheartedly. As it 
currently stands- I would break my monitor trying to read even a small number of 
them. 

I can definitely see potential in the analysis features of this software- as long as the 
data is accurate and timely- but the performance appeared a little slow to me (not 
knowing what's under the hood). I understand I could create smaller sets of data- 
but this might be counterproductive and not provide the big picture depending on the 
intent/scope of this software. 

It was adequate. Somewhat slow- but that is understandable considering the amount 
of data involved. Functionalities are quite intuitive. Inability to zoom into charts is 
slightly inconvenient. 

No. It's jittery- unresponsive. Takes forever to load information. The user help is 
not very friendly. This software needs serious help. 

While the goals of the software and tools available were helpful- I felt that 
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Q: (Satisfaction_7) Were you satisfied with the performance of the software? 

Please explain. 

functionality could be improved on the map interface. It is slow and clunky- with 
non-intuitive pan / zoom / selection buttons. Support for mouse wheel scroll for 
zooming would be very helpful- and reducing the delay between data selection and 
analysis would be helpful as well. 

The software yes. The chart interpretation no. 

Not really- it's very slow to respond to commands. 

Yes very satisfied- I accidently marked answers strongly disagree when I meant 
strongly agree. This software is user friendly and not difficult to navigate around in. 
The time required to load and unload shapes had quite a bit of a delay. 

The software has some good output. The type of charts needs to be looked at- some 
charts are better than others and easier to read. The speed of the program was a bit 
slow at times generating charts. 

Series of clicks- left or right and where you had to click was not straight forward 

I assume that the graphs can be made larger. Latency of data display is not good. If 

I was using this in an operational environment I would have to get up and smoke a 
cigarette to kill the time while waiting for data to load. If I did not have a smoke I 
would just beat my machine with a hammer. I'm confident that once the data is 
displayed that I would be able to use the tool positively. 

Table 57. Table of free-form responses to question Use_2 for all 24 subjects in session 
three. Comments taken directly from the EyeWorks software with no 
modification. 

Q: (Improve_3) How would you improve DaViTo? Please explain. 

There is something off with the mouse selection of individual territories associated with 
lag and the need to deselect an area prior to the next selection. Perhaps a "waiting" 
icon (hourglass?) could be displayed to let users know when to wait prior to entering 
the next command would be appropriate. 

More hands on training. Executing the tasks that are necessary to use the software 
efficiently. 

I would add chart generation to the toolbar. 

Improved lag on map controls- status icons for processes. 

Progress icons- more training 

bind the most commonly used geographic analysis software out there that resembles 
DaViTo and organize the GUI to resemble that as closely as possible. 

See previous comments. 

Improve the lag problem. 

I would improve DaViTo with a tutorial by an onsite representative to teach the 
functionality of the program. I would also leave a basic video tutorial to ensure that the 
use of the system was driven home by the user. 

If you hover over an item it should pop up "quickly" telling you what it is. It seemed to 
be pretty delayed. I was still doing a lot of 'trial and error' while manipulating the 
software in my third session. I'd also have a user-friendly help section that we could 
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Q: (Improve_3) How would you improve DaViTo? Please explain. 

click on. Additionally- I'd like to see a way to personalize the display buttons etc. for 
use. That way I could put things where I'd like them. 

Make it faster. 

Improve user interface issues essentially to make it more intuitive for the user; improve 
the flexibility so the user can modify output; and fix system latency. 

Improve the data set loading issue. Other than that- the tool was excellent. 

Make the graphs easier to read 

Improved charts- fix the lag (see previous comments) 

Navigation button could be renamed 'zoom box' or something similar to avoid 
confusion with the map navigation button.- user could be given more control of the 
charts (e.g. zoom in to some portion of the data)- loading time if possible (e.g. when 
loading 

Make it more user friendly and responsive. 

Comments provided in previous questions. Better maps and response times¬ 
es sentially. 

Spend more time accomplishing the tasks and improving the user friendly charts. 

Speed it up and improve the graphical outputs 

Figure a way to remove the delays when loading or creating shapes 

Again- improve some of the charts- I thought the cluster charts were not good visual 
representations of data. 

Charts 


Table 58. Table of free-form responses to question Improve_3 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


Q: (Improve_4) What was the most frustrating feature of DaViTo? 

The lag. 

On screen help. I should have used the help button more. This would likely be 
eliminated by more training. 

Sluggishness and chart generation. 

Lag 

Not knowing if it was frozen or thinking 

Finding the chart option for Kenya. 

Chart output was difficult to match to months and difficult to add up events. 

Lag time in loading things and lag time between selection and reaction. 

Interpreting charts and lag time for map loading 

It seemed really slow. Lots of delay loading maps and right clicking. Sometimes the 
delay would be counterintuitive because I thought I was doing something wrong- but 
in reality it was just slow. 

It was slow. 

System latency 

Data set loading time. 


132 







Q: (Improve_4) What was the most frustrating feature of DaViTo? 

Graphs- graphs- graphs 

Lag time when selecting maps/data sets 

Load times 

Slow and unnecessarily complicated to navigate way around data. Confused on where 
to look to load data...took too much time. 

Slow map response times. 

Interpreting the charts. 

Sluggish responses 

The delay experienced when creating or loading a shape 

The slow time of generating charts and trying to read and interpret the cluster charts. 

Which mouse button to use and when in what context 

Latency of display/function execution 


Table 59. Table of free-form responses to question Improve_4 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


Q:(Improve_5) What did you most like about DaViTo? Please explain. 

Like ARCGIS- the ability to display data on a map is very useful. I'm not sure how 
this software would be better than ARCGIS- though- and that is a widely 
disseminated and developed product that is significantly less buggy. The ability to 
predefine areas that a user could then open- view- and process for analysis makes it 
possible to share work without having to recreate it while using it for different 
purposes. That could be a great potential time saver that would also reduce 
confusion between units by sharing world views. 

The set of the map with controls to the left. Seemed much more logical and I could 
see more of the area I wanted to see. 

Visual display of event data and layered maps. 

Good visual representation of geographic data. 

I like maps 

Overlay of chart data by geographic region so you can eyeball outliers/trends. 

The general interface. It was intuitive. 

Simple. 

I liked that basic graphing information could be loaded to represent trend data for a 
given location. 

It seemed simple. It didn't look overwhelming or intimidating. But in the end 
simplicity wasn't the best way to go. 

Nothing. 

That it is open source and can be improve 

The ability to populate different districts and boundaries for different countries/areas. 
The ability to select a specific region- and then it provide detailed charts on the data 
is a great benefit as well. 

Ease of use 
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Q;(Improve_5) What did you most like about DaViTo? Please explain. 

Fairly intuitive. _ 

Gives a good view of the whole data set at a glance _ 

Clicking on a data set and seeing data _ 

Integrated data analysis. _ 

Navigation of the map and the ability to bring up the loaded data about the 

provinces. _ 

Nothing really excited me _ 

Easy to use _ 

Simple- easy user interface. I am not usually fast with learning computer and/or 
software programs and the fact that I learned some basic functionality with this 

program in a short amount of time speaks well about its usability. _ 

Graphical display _ 

Ability to visualize data _ 

Table 60. Table of free-form responses to question Improve_5 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


Q:(Improve_6) What would you change about the color scheme / page layout? 

Please explain. 

Slightly greater contrast might make it more comfortably viewable- though it was 
sufficient. 

Eor daytime use- the color scheme was sufficient. The major change would be for 
detailed use having more area relief. This would be useful both on the move and in 
planning. I would know how to plan for movement times. 

Nothing 

Nothing 

Nothing 

Nothing 

No. 

Nothing. 

I would add actual colors- vice the drab grey. I would also have areas of high 
importance be color coated differently than the surrounding areas to represent their 
importance. 

Too bland. I don't need a lot of colors- but if I had to look at it all the time I'd like it to 
be easier to customize. The default colors scheme/page layout should be a little less 
bland. I get bored just looking at it. 

Less shades of blue. 

Given that used the tool for less than 2 hours- I cannot properly recommend a color 
scheme or page layout 

No 

Nothing of note 

Kind of bland compared to most applications. A little more vibrant color scheme and 
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Q:(Improve_6) What would you change about the color scheme / page layout? 

_ Please explain. _ 

larger fonts might make it more attractive. Compared to dynamic mapping software 
(like Google Earth) - this is at the lower end of the spectrum (not like Apple user 

interface that "slides" the map along and incorporates inertia). _ 

It’s fine the way it is. perhaps the default color when loading a data set could be 

something of high contrast- e.g. red- to start with (instead of dark grey) _ 

Use colors to depict certain areas on the screen for better navigation. _ 

Layout was good- but utilizing a 3d map like Google Earth would be far more intuitive 

and helpful. _ 

No issues. _ 

Nothing _ 

Nothing _ 

None _ 

Nothing _ 

I think it is fine. _ 

Table 61. Table of free-form responses to question Improve_6 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


Q :(Improve_7) How would you improve the mouse interaction with the 

software? Please explain. 

I don't believe there were positive indications of mouse selection or actions associated 
with some of the functionality. When specific selection is required- definitive 
highlighting might be possible. 

Nothing. 

Unknown. 

No improvement necessary 

Right click on map gives all operations 

Nothing 

I wouldn't. It was fine. 

Nothing 

No comment 

Just make 'right clicks' faster. Also- give more ways to navigate with just the mouse 
like panning. I think there is that option but it was too slow. 

I wouldn't 

It seems that the mouse interaction works properly 

The mouse is fine. 

It worked fine 

Nothing - works just as expected with left and right clicks for primary and secondary 
actions. 

Mouse clicks for the distance tool was a bit confusing 

Scroll map while depressing the left mouse key 
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Q :(Improve_7) How would you improve the mouse interaction with the 
_ software? Please explain. _ 

Integration of map navigation into mouse buttons. _ 

Change how the chart menu option is brought up by clicking on the location. Seemed 

to take numerous attempts for the menu to pop up. _ 

No issues with the mouse _ 

None _ 

Table 62. Table of free-form responses to question Improve_7 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


Q :(Improve_8) How would you like to see the software employed? Please 

explain. 

If this software could be adopted by Intel and ops analysis groups- it would be 
awesome to share prepared data. That said- it would have to be unilaterally adopted 
within the joint construct or it would be a waste of time because data portability would 
suffer. 

In TOCs to assist planning. 

Resource utilization based upon event data. Allows focused use of limited resources. 

Data analysis for counter piracy operations 

Intel- mission planning 

I could have used this to track my squadron's lED detection performance. P-3C 
missions up and down Iraq's supply routes report hundreds of "hot spots" and some get 
investigated while some do not. Those that are investigated may be lEDs- but many 
more are false positives. The trends are somewhat geographically related- but the data 
isn't easy to visualize. Better data tracking would be required to use DaViTo for this- 
but if you had it- you might get the commands to track data appropriately. 

I can think of several uses. It would be useful for anything from intelligence events to 
liberty incidents to logistical tracking. Wide variety of possible uses. 

Not sure. 

I know this is mainly an Army software tool- but if there could be incorporation of it 
among other branches of military it could broaden the horizon of data available. 

I'm not sure... 

No idea. 

Currently my experience with the tool is exploring event data but this tool could easily 
be modified and employed for sustainment and logistics personnel. 

In training and field environments (Afghanistan- of course) 

I suppose if it were made available for basic research. I had never heard of it before. 

Integrate this software into CPOE rather than as a stand-alone application. The 
features I've seen indicate a focus on analyzing data sets for trending that could be 
applicable to both the tactical and operational levels- so it would be better to have a 
shared platform for looking at the same data. 

Probably useful for the OR and data analysis guys. 
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Q :(Improve_8) How would you like to see the software employed? Please 
_ explain. _ 

Used for trend analysis in order to make better decisions at company level and higher 

Basic analysis of available data for military needs. _ 

Useful as a planning tool for events in areas being entered. Could give a visual 

indication of the activity expected to be encountered. _ 

Could be used in a HQ to conduct trend analysis for SIGACTS _ 

Would be useful for land and see navigation _ 

It seems it would be more useful for land based forces. I serve on Navy ships- and I 

do think it would add much value there. _ 

I have no reference _ 

Would be useful in organizational headquarters first in a garrison setting and then 
deployed. Garrison use would ensure that both the tool and the user were familiar 
with the system and the interpretation of the data prior to operational employment. 

Table 63. Table of free-form responses to question Improve_8 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 


Q: (Visual_2) What more would you like to see in the output of this software? 

If you could link the data through R- Jump- or Excel - the displays might be more 
consistent with some of the data display tools we all have to use anyway (especially 
when constrained to an NMCI machine). Perhaps both cumulative and incremental 

charts could be displayed simultaneously (to alleviate confusion). _ 

Expanded graphing such that you could zoom into a graph and discern individual data 
points - including scales that would be appropriately detailed to discern where the 

data points sit. _ 

Just a clear definition of whether events are independent or cumulative. Also- why do 
the events have to be stacked or put together? This is not provide me with more 
information. If I just wanted shootings and not lEDs- why would I get all the 

information all at once? I personally want to be able to tailor the information. _ 

User modifiable graphs. _ 

Statistical analysis of different portions of the data- i.e. analysis of all lED attacks for 

a particular month over all of the provinces. _ 

Data should be more clear in its representation. Possible hovering mouse shows 

nearest data point value _ 

More options for graphing than are shown in the questions (assuming that's all there 
is). The axis labels are confusing and some of the cumulative line graphs are 

misleading because of their alignment. _ 

I addressed this a bit already. Make the manipulation of the output more intuitive. It 
should be able to be manipulated by right-clicking or with first-level menu options. 
Perhaps a GUI where the user could ask specific questions- like posed in this 
experiment- and the software displays the correct answer in addition to the graph. 
This could be done from pull down menu selections. Many current financial websites 
do this for the customer. 
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Q: (Visual_2) What more would you like to see in the output of this software? 

Basic examples on how to read the data would be beneficial. 

A way to hover over the graphs and get data points (amounts) would help. Or be able 
to click on the graph and get the actual numbers instead of generalizations done with 
the graphs. 

For big graphs it is hard to see where the values line up on the y -axis. If the axis 
could be slide to the right so it is next to the data that one is interested in it would 
make interpreting the data easier. Also- it would be nice to zoom. But all of this will 
be pointless if the program keeps running so slow. Instead of redrawing all the lines 
frame cycle- like I know this thing is doing- why not write the data to a background 
buffer. Stuff can be put over this buffer but it doesn't need to be recalculated every 
time. 

Essentially improve flexibility so the user can modify and edit output similar to 
capabilities provided by other tools including excel. 

Nothing more to add; just improve the output of some of the charts- meaning the 
actual display (size in general). 

Make the graphs more intuitive- as the software itself is. 

Have the ability to toggle quickly between the types of charts to pick which one 
presents the most clear summary of events for which you are analyzing. It would be 
interesting to add some real analysis features such as extrapolation- prediction- and 
so forth to leverage statistics- but I'm not sure how difficult that would be. 

Bigger charts (when required) 

Graphs that a human can understand without performing some kind of advance math 
problem. 

Explanations available for users that are not proficient with basic statistical analysis 
would be useful in the output. 

More user friendly charts. 

The graphs should be tailored to the query. 

Nothing else 

Easier to read charts. 

Pie charts- for one 

Cluster column format needs to be spread out. Fine the data up with the grid line 
references to make it easier to follow. 


Table 64. Table of free-form responses to question Visual_2 for all 24 subjects in 

session three. Comments taken directly from the EyeWorks software with no 
modification. 
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APPENDIX E. TASK WORKSHEET 


Appendix E contains the list of tasks for the final session, as shown in Table 65. 
This form was used by the experiment session proctors to track subject performance. The 
mouse clicks and times for each task were recorded by EyeWorks. The proctor 
monitored and verified the status of satisfactory task completion. 


Participant Number 

Task 

Clicks 

Time 

Completed 

Y/N 

1.) Open WITS.csv 




l.A Left click + button in Data Set 




l.B Left click WITS.csv 




l.C Left clicks WITS.csv open button 




l.D Left click load button 




Total 




2.)Load Kenya_Roads Shapefile 




2.A Left click + button in Predefined 
Polygons box 




2.B Left click Shapefile Kenya 




l.C Left click open 




Total 




3.) Go to Africom view 




3.A Left click Views 




3.B Left click Africom 




Total 




4.) Create a rectangle chart on the data Point 
in the southwestern most part of Kenya and 
display the chart 




4.A Right click on screen to create 




4.B Left click on rectangle 




4.C Left or right click to draw rectangle 




4.D Left click on screen 




4.E Right click on rectangle 




4.F Left or right click on "Display Chart" 




Total 
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5.) Close and Delete Chart 




5.A Left click red x Button 




5.B Right click on rectangle 




5.C Left or right click on delete chart or 
delete all charts 




Total 




6.) Hide Map controls 




6.A Right or left click on control tab 




6.B Right or left click on hide tool panel 




Total 




7.) Navigate to Iraq without map control panel 




Option 1 




7.A Right or left click control Panel 




7.B Right or left click on pan 




7.C Navigate to Iraq 




Total 




Option 2 




7.A Right or left click views 




7.B Right or left click on Iraq 




Total 




8.) Go back to previous view 




8.A Right or left click control tab 




8.B Right or left click display tool panel 




8.C Left click go back previous 
projection icon 




Total 




9.) Remove all the Data sets and Shapefiles, 
load the lndure_Demo data and 
afganistan_provincial_boundaries Shapefile 




10.) Now please answer the following 
question: what were the totai events for lED 
attacks for 2010 in the Northwestern most 
provincial boundary? 




11.) Does it appear that lED attacks are on the 
rise from 2006 in the same provincial 
boundary? 




Total 





Table 65. Worksheet used by experiment proctors for data recording. Includes all tasks 
for session three. 
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APPENDIX F. APPROVED IRB DOCUMENTS 


Appendix F contains the IRB approval cover sheet and author generated IRB 
documents. This includes the consent form, call for participants, demographic survey, 
and question list for the final survey. These are the approved documents from the IRB 
for use in this thesis. 
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NuvmI Postgraduate School 
Consent to Participate in Research 


latmdBctwn 

Yoa are inviaed to {tadicipate in a reicarcti sUady tilled Usability Testing and Woricflow AimI>'(is ot the 
TRADOC Data VisualiTotinn Toot (DaVITo)" This research is focuted on evaliialing tbe usability Mtd 
wocifVns ot the DaViTo tool, with an aim of imorovcnictM. 

Purpoor; 
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are intendod to improve the workflow, usability and vawal claril}' of DnViTo. 
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patticipnni will be asked to compkar a dnnopapbic quesiioMMire and creak a profile in the eye tracking 
sofimare, then match a familiarization video of Uw OiN'iTo software During the seoond Iraintog and 
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appraximalely 3 to 4 hours. Any infeemation will be used only for this research endeasor. 

Voluiilary Nature of the Study 

Y our partKipalkin in this 9ud>' Is slnetly voluntary. If you choose to participalc yssu can change yxiur mind 
tt any time and withdraw from the study. You m ill anst penalized in any way or lose any bcneflls to 
which you mould ctherwaic be entitled if you choose not to pankipiie in this study or to withdraw. The 
ahemalise to particqiMiiyg in this research is to not pwlictpaie in title research. 

Polraliol Risks and Discomfoiis 

1he potential liiAs of pwticipeimg in this study are minimal You will he sitting in front of a large 
monnur using a coav'cntional mouse and keyboard to mleract with the DuVtTo sofimare. The eye scan 
casnera systems have inftannl lights associated mmh them These are no more harmful than normal room 
lighting There mill be many sale guards in pbee to protect the patticlpant's ccnfidencialily; however, 
llierc is always a minimBl risk that there migtu be a broach of confidentiality. 
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Aallvipaivd Bvntftls 

AnikipiiUNl btoelib rrom Ais itad> ore to improved lh« UHbili(> for the DaViTo software, and provkle 
VKuali7atton data in a clear and mderstandable wav for Ac dccifion iwdicr. Ihera it no direct benelit to 
yu« for participating. 

CoaipMiidicitMi fur Parlkipatiaa 

)vo taiigibk coonpensation will be given lo the paniciftaM. 

CanfMcatiality A PTivac)i Aei 

Aay infonnatKW that i» obtainad during Ais study will be kept oonfidcMial to Ilia full anient penniited by¬ 
law. All elTom, wiAin reanon, wiM be ntade to keep your persmal infbmiation in your research reocird 
confidcaMial but total confideatialily caanoc he guaranteed 

AD surveys will be proaecoed by a pasoword revurity and at a secure NPS terminal If it Is aecasHiry to 
prtna any wrveys. all personally idimliflable Information (Pll) will he scrubbed so the paiticipam camiol 
he identified All printed surveys will be stcaed in a tacked drawer in a secured lacilHy when not in use. In 
the final publication, no names will br used for privacy rciucins and hecaiase it adds ao vahw to the 
reaeurch. 

All references to data collecied will he mode anonymous. Your isame will he encoded as a purlKipail 
number. Only- principle invesligalors will have acoeus to Ais key-that transAtes an identification maniber 
to your name. 

Paints of Cnatact 

If you have arty questions or comments about the research, or you experience an injury or have questions 
aboM any disoamfotts that you eapent nee while taking part m Ais study please contact Ac Principal 
Investigator. Dr. Ji Hvun. 831-6$6-3O0-t. iv an I njincs.adu Questions about your rights as a rcseirch 
stdijcct or any otlier eoncems may be addressed to the Navy Postgradutae School IR0 Chair, CAPT John 
Schmidt, («J I 

Statement af Cunaeal 

I have read Ae nfonnatioii prov ided above. I have Awm given the oppertunily to ask questions and all A« 
tyuntiions have bten answcied to my satisfaction I have been pros ided a copy of Ais form for my records 
and I agree lo participate m Air slody. I understand that by agreeing to participate m Ais research md 
signaig Ais form. I do not waive any of my Agnl rights. 


Participant's fiignalure 


Date 


Researcher's Signature 


l'pdaicdon:4/|Q/|2 


Date 


IPS III 

APPROfEO 

APR t1 
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DCMOGRAPMC SlMVrv IM (KAMOOM 4 OMITS);_ 

1. Ple44« «nr*r your tge in years: 

2. Please indicate your militatv icmce brancti IChack Ona)- 

□ A«A«Y dNAVY a marine corps O AIR FORCE Q ACnREO MH/TART 

U mnRNATIONAl. MMJTARV MEMBER 

) Is Ertflisli yosir primary laniuace? (Check Onel □ YES □ NO 

4. Please aidicatc yoia military grade le,g. 0-2. 0-J, 0-4. etc I - 

5. Please mdicate H you are afTiliated wrrth the development of the OiaVITo software; (Check One) 

□ YES cNO 

6- Are you now or have you ever been empfoyed as ars analyst’ (Check One) d YES o NO 

7. Have you ever used the OiaViTo software before you participated at this study; (Crete One) 

□ YES oNO 

8. Check the most descriptive urut of measurement that describes the amount of evperleisce you 
have with computer software: 

□ None 

a Novice (word processirtc, errtalC web surfing 

a IntemvaiMata user (some programminj in at least one python. C, sava. ruby. jmp. ate.) 
c CKpert (Computer Science backgroiasd, or can create complex procrams) 

9. How many hours per week do you spend on a computan (Chack Onal 

O 0-5 hours CB-IO hours O 11-15 hours a 16-20 hours a21rhours 

10. Oo you have 20/20 valon, or corractable to 20/20 vision? qVES UNO 

11 Are you color blind? oVES cNO 

12. Hava you ever used ARC CIS or any other geo spatial mappinR software: (Check One) 
dYES oNO 

1). What operatinc system do you prefer? 

□ Mac Q Wirsdows a Other 


IPS III 

xM'fViiie 


IXPIKS 

SEP SO 25tt 
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Quantitative & Subjecttva Quatllont 


e«<t iufvev 


Pteaie check or answer the foHowtiig questtons about the experience you had usasf the DoViTo 
Mitware. Please use the follownrtg scale for your responses: 


Strorgly Disagree 


Neither 

kgree/'Oisaiiree 


Strongly kgree 

i J 

1 2 1 

i_ _J 

4 

L_ 5 _ 1 


Satisfaction (refers to; perception, feelinc. and opinion^ 

1. I feel confident in my abdKlas to use OaVTTo for bask analysis: 

□1 q 2 a) 04 d5 

2. I think I can succetSluVy hnish basic tasks without the user's manuaf 

Dl 02 03 04 CS 

3. I think I can successfully finish compiei tasks vwih the user's manual: 

□ 1 02 o3 04 gS 

4 I think this software would add value to missions that I have performad- 
Dl o2 d3 04 o5 

5. I would recommend this software to my command: 

□1 02 Ql d4 dS 

6. I feel comfortable using oaVITo wHhout any assistance: 

Gl 02 o3 04 oS 

7. Were you satafied with the performance of the software' 

(Written response) 


Visual (refers to: represenlatien of output): 

1 The way in which output data is dnpiayed b dear to eitarprer 
01 02 03 04 dS 

2. What more would you like to see In the output of this software? 
(Written response) 

3. Graphical Representatnn queston 1 
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Qujfitiutw* & Sub)«>cnu« QuMtions 


4. Graphical Repratcnrtallon lyuauion 2 


S. Graohical Reprevpiitation qiwitum 3 


6 Graphical Reprcientation questiaii 4 


I hCfirl , 

* \. 0 ^ 

' ^ ‘ r « rr^ 

■ oCOJ^f SUi'l 


UsafuliMu (nafan to: ahlUty to achieve coalx, ease ol use, and w^lngnet* to uee^; 

1. The tasks were easy to complete: 

□1 02 03 04 aS 

2. What was the most dimcult task? 

(WrNten Respome^ 


3. The termiooloitv for buttons and commands was clear and undersiandabie; 

ol 02 ol o4 dS 

4. tf I were an analyst I would use this tool tor ReM analysis: 

ol 02 03 tt4 as 

5. A consputer novice would be able to use this sohware with the irainint provided: 

ol 02 03 a4 aS 

6. A computer novice would be aWe to use this software wwh |utt the user's mansial: 

nl 02 o3 o4 aS 

Efficiency (rotors to: quickness to accuratefy accompllth foal, layout, time benchmark and naw«atlon>: 

1. The organuatlon of the screen matched my expectation of button locatton 
t:l d 2 03 n4 oS 

2 SlmdUr icons and activities are consistentlv placed: 

Ol 02 d3 q4 u5 
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QuMitiUtm & Subjective Questons 


3. Due to my exp«ctenc« wlttt computers, ivtviitation of OaVtlo was intultfv*: 
al d2 d3 o 4 nS 

4 Was the visual representation easy to understand? 

(Wrtnen Retponee) 


5. The color scheme for the software was appropriate; 

Dl 02 o3 04 aS 

6. The tert wtthm output graphical representations was easy to read and match to data: 

al d2 n3 04 oS 

7. The software header bars and dropdown lists nsade navigational tense 

Ol o3 Ol o4 qS 

a. The software was stable, no freeiing: 
al a2 a3 o4 o5 

Effactiwenass (refers to: software behaves as expected, workflow, error rata, mousa dkk banch 
matkj: 


1. I understood where I was at in the software as I progressed through a task: 

□1 02 03 04 aS 

2. The mousa eiteracted with the software as I expected it to: 

a 1 a2 n 3 o4 □ 5 

3. Was the software simple to use? 

(Written response! 


4. I was satohed with the reaction tone of the software: 
ol 02 c3 a4 CIS 

S I felt there was an unnecessary amount of mouse dickirig to complete the given task: 
Cl o2 a3 04 o5 

6. I made fraquanc arrors trying to complete the given task: 
al cs2 al 04 dS 
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QuMitItatMie A Subjective Questiora 


Leamabiiitv (refers to: compeurtce, trelning lime, confidence, relearn an aciMly): 

1. I think moa peoplecwouU Ivjre out how to use this toftvMre wKh the training aids provided: 

□1 a2 □) 04 oS 

2. I Ihink moJI peopfe could figure out how to use this software with just the user's manual: 

Ol 02 03 d4 oS 

3. No more training is needed to effectively use this software 

Ql 02 03 04 o5 

4. If given these usiu in one month to compfete again. I feel I could erecute them: 

cjl 02 03 04 dS 

5. I could show someone who has no eiperience with OaViTo basic hiiKtIonaItty 

Ol 02 03 o4 05 

improvemants (refer to the users ssvcestMns): 

I User's manual is easife leiderstandable 
01 02 u3 04 oS 

2. More traliwig aids are needed for me to effectwelv learn to use the software. 

□ 1 D 2 o3 o4 0 5 

3. How would you improve DaVITo? 

(MNitton responae) 


4. What was the most frustrating feature of DaViTo? 
(Wrfnen responscj 


S. What iM you like most about OaViTo? 
(Written rMponsa) 


6 What woidd you char^ about the color scheme? 
(Wiitten response) 


7. Mow would you Improve the rnouse interaction with the software? 
(Written response) 
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Qutnlitative & SubfcctM Quottiont 


8 How would you Utc to Mtitw {o4tiwar« «mplov«d? 
{Writtvn response) 
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Graphics Questions 


o 

Oo you know wh»t this kon means? 

OOrawIng oPen DOotance a Navigation a Gestures o Go forward to nest proieclion 

13 Go hack to previous protection oMapNan^tion o I don’t know 


Oo you know what this kon means? 
a Drawing □ Pan o Distance □ Navigation n Gestures o Go forward to next projection 
O Go hack to previous protection O Map fuviBation o I don’t know 



—I— I Oo you know what this kon means? 

O Drawing o Pan oOisUnce a Navigation a Gestures o Go forward to next protection 
o Go back to previous protection o Map Navigation □ I don't know 



m> 

Do you know what this kon means? 

□ Drawing o Pan □ Distance cNavigatasn o Gestures a Go forward to next projection 

□ Go back to previous pra|ectton □ Map Navigation a i don’t ktvjw 

ss 

Do you know what this kon means? 

PI Drawing a Pan □ Ontance □ Navigation □ Gestures □ Go forward to next projection 

O Go back to previous projection □ Map Navigation Of don’t know 
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Goiihacx OuKtkMU 


$ 


Do you know 

□ Drawinc □ Pan 

□ Go back to previoui 


what thtt icon ni«>an&? 

a Diitante n Navigation o Cotturat □ Co torward to no»1 projetlion 
projaction G Map Navigation o I don't know 
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